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Unless he Is well organized and very persistent in following his strategy, he mgy lose track'of his strategy. Even if he 
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finding the trouble. Based on this rationale, it was concluded that previously developed symbolic tests such as the 
tab test did not provide such clutter. It was hypothesized thst symbolic substitute t^sts could be developed thatc 
would retam a large amount of realistic tas\ "clutter" and that such tests would have higher empirical validity than 
pr^vfously devdoped symbolic tests. | 
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In^ this effort, a battery of symbolic tests was developed irwluding a companion symbolic test for each of the job 
£ ag^ivities for which a criterion referenced iTPT had previously been developed. Based.on two limited validations, all 
thegraphic symbolic tests, with the. exception of the symbolic test for soldering, mdicated sufficiont promise to 
. justify further consideration-and refin^ent. All of these promising symbolic tests should be given more extensive 
validations usmg larger numbers of experienced subjects. The validation of any such symbolic test requires the' 
administration of a companion JTPT as a validation criterion. As a result, a validation is an expensive process in 
terms qf equipment and experienced manpower. The troubleshooting symbolic test's requiro^the most extensive 
refinement. Several suggestions are made for improving their empirical validity. ' * , , 
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SUMMARY 

• ♦ • %' 

Problem ' " ' ' .A 



An in-depth review of the literatuj^ reported in AFHRL-TR* 74-57(1) of this series o^ documen 
strongly reiterated the fact that paper ana pencil tests of job knowledge and electronic tlieory tests have 
very poor criterion related or empirical validity with respect to the ability of electronic maintenance n\en^ 
for performing their job tnsks. As a result, a battery of criterion referenced Job Task Performancp/Tests 
(JTPT) was developed an^'tried out and was reported in AFHRL-TR>74-57(1I), Part I. Tlie battery included ' 
Jests for tlie various job activities performed by electronic maintenance technicians such as checkout, 
align/adjust, remove/replace, soldering, use of general andvspecial test equippient and t^oublesliooting. 
These tests require the test subject to perform such activities using the actual equipment found in tlie job 
environment. "Past experience witli tests similar to these criterion referenced JtPT has indicated tliat even 
thbugli criterion referenced JTPT were recognized as being superior by many training people, paper and 
pencil tests were substituted becaiise tliey were more easily and cheaply developed, and administered. The 
JTPT developed required the* test subject to use actual-equipment and tliey can only be administered to 
very small groups, in some cases, to only one subject. If empirically valid symbolic substitute tests could be 
developed, they \Vould be much* more economical to administer. As a result the reviews of previous 
attempts concerniitg tlie development of graphic symbolic substitute tests such as the tab tests, it was 
hypptliesized that previous attempts haH weaknesses in realism tliat possibly could be rectified and thus 
improve their empirical validity. . • * ' 

Appcpach and suits * 

The successful accompllsliment of most maintenance task5 must follow a main line procedure or 
strategy. But tliis main line usually is "cluttered" with distraction and subprocedurcs which interfere \yith 
the accomplishment of the main line procedure or strategy. For^example, when troubleshooting, the 
technician must usually interrupt his strategy ^several' times to set up his test equipment, and obtain check 
point infoniiation. Unle^ he, is well organized and very persistent in following his strategy, he may lose 
track of his strategy. Even if he stays on tlie , main line, he may gather faulty information from his test 
equipment which will prevent him from finding the trouble. Based ^i tliis rationale, it was concluded that 
, previously developed symbolic tests, such as tlie tab test, did not provide such clutter. It washypotliesized.* 
that symbolic substitute tests could be developed that would retain a large amount of realistic takk 
"clutter" and that such tests would hay« Iwglier empirical validity tlian previously developed symbolic tdsts. 
In tliis effort, a battery of symbolic' #sts was developecl including a cl)mpanion symbolic test for each of^ 
the job activities for which a crHoJrion referenced'JTPT had previously been developed. 

These symbolic substitute tests were validated in a limited^llanner against co^ipanion criterion 
referenced JTPT at Altus AFB, Oklahoma, using 14 novice technicians as test suhjects^Novice tecl?nicians 
were utilized because prevlously»developed plans for use of experienced technicians did not materialize. The 
result of this first validation was tliat all of the symbolic tests indicated higher corTelations^than obtained 
from tlie previously reported studies, with the exception of the symbolic tests for troubleshooting at^ 
soldering. / ^ 

With the limited remaining available' funds, it was decided to concentrate on improving 
troubleshooting symbolic tests. It was felt tliat ^nost of the difficulty with the troubleshoming symbolic 
tests during the first validation were due to tlie»inexperienced subject^ and to the niannel in which test 
equipment symbolization was presented. To present realistic "clutter" in the original troubleshootings 
symbolic^tests, a pictorial representation of the test equipment had been prepared for each test point for 
which information would be required. Two problems develo()ed. fTrsZ/the bank of pictorials was placed.in 
a book and the Test ^Mniinistrafor was required to search this bank when a ^pst subject required a bit of 
infg^mation. This proved to bp an intpossi!)l€ task for the test administrator. Second, the number of 
()ictorials in the bank was insufficient to meet tlie needs of the novice subjects used in ttie validation. To 
overcome these 4)roblems, an attempt was made to consolidate the test point information into a matrix . 
fonnat with printed values for voltage and resistance measurements and small waveform drawings for 
oscilloscope readouts. These modified symbolic tests Wfere validated at Langley AFB, Virginia,.and Little 
Rock AFB, Arkans^. using a totSl of 15 experienced technicians as test subjects/ The resulting symbolic 
tests indicated a high empirical correlation t6 the criterion JTPT for the black box or chassis level of 
troublesliooting, a fi^r correlation for tlie stage level and an extremely low correlation for th^piece/part 
tevel. A secondary finding of this second validation was.tliat the sample of experienced tfechnicians.used for 



this validation could not trouble^oot very weU to the piece/part level. They identified only 33^percent (5f 
defe"ctivb4>iece/parts, ^* 

Occlusions \ • ^ 

\. Based on two limited validations^tall pf the symbolic tests with the exception of the one for 
soldering indicated sufficient promise to justify further consideration and refinement. • 

^^2. All of these promising symbolic tests should be given more extensive' validations using larger 
numbers of experienced subjects. The validation of any such symbolic test requires the administration of a 
companion JTPT as a validation criterion. As a result, a^validation is an expensive process in terms of 
equipment and experienced manpower. Until su^h support car^be guaranteed, no further work on symbolic 
tests should be pursued. ' j 

J. A validation of troubleshooting symbolic tests requires, special considerations. Before 
administering the modified troubleshooting symbolic tests again^ it should be ascertained that the test 
, subjects can use their test equipn^ent adequately. This prerequisite may result in hi^e^^ correlations at the 
stage -and piece/part levels of troubleshooting. ^ 

4. The modifm troubleshooting symbolic tests are a giant step removed from the original 
hypothesis concerningfhe requirement for realistic "clutfer.".The action recommended in paragraph 3 may 
not be sufficient to raise the correlations of the piece/part symbolic test to an acceptable level.^'A system 
for tlie randoHY access projection of test equiprnent pictorials of the type included'in tlie original version of 
the troubleshooting symbolic tests should be developed. A* matrix system, similar to that4^sed for the 
presentation of printed voltage and resistance information in the modified troubleshooting Syrtibolic tesfs," 
could serve as the basis for a simplified ipdexipg system. This could be accomplished by substituting 
projector access nuntbcrs for the printed voltage or resistance numbers in each cell of the matrix. 

5. The fact that the experienced subjects, used in this project, could not.troubleshopt well to the 
piece/part level is another bit of hard data that suf^^orts^othev' available hard data which indicate that a 
widespread weakness exists in this area. / . «^ 
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EVALUATING MAINTENANCE PERFORMANCE: 
THE DEVELOPMENT OF GRAPHIC SYMBOLIC SUBSTITUTES 
FOR CRITERION REFERENCED JOB TASK PERFORMANCE TESTS ' 
FOR ELECTRONIC MAINTENANCE • 

I. INTRODUCTION 
Background j . . ' 

V Volume I (AFHRL«'f^74-57(I)) of this series of documents provides an analysis of the effectiveness 
of current measurement practice concerning electronic maintenance tasks. One of the common fipdings of 
the many studfes considered in that analysis was that the empirical validity of the paper and pencil tests 
with respect to performance is notable for the low conrelations found and for the frequent use of dubious 
criterion measures such as supervisory rating oCjob performance. Yet, test validity is frequently assumed 
and advancement in grade is partially based'on these .dubious test result^ Another conclusion of that 
analysis was that little systematic^ work.had been reported on attempts to define the scope and to improve' 
the effectiveness of criterion referenced Job Xask Performance Tests (JTPT). Among the recommendations 
made were that: . . ' " - , • 

1. A* battery of JTPT Jlould be developed that would independently measu^ie^ach type of job 
activity; namely, (1) ch'eckout;(2) remove and replace;(3) use cff test equipment; (4) us^*pf hand tools; (5> 
/align, adjust, and calibrate; and,(6)troubleshoot. ^ 

2. An appropriate scoring sche;ne should be developed foil^^of these'types of job activities. 

3. More experimental work should l>e accomplished aimed at developing and refining symbolic 
electronic equipment troubleshooting tests. 

.Under a previous contract with URS/A^trix Research Company (F33615-69C1232), the Air Force 
Human Resources Laboratory (AFSC) si^Mrted the development of the recommended JTPT. The 
development and tryoul of a battery of sif^ JfPTls described in Volume II of this series of documents 
(AFHRLTR-74-57(II)). As a result of the work" reported in Volumes 1 and II and the insight gained from 
this^work, certain conclusions can be stated that effect the-development of symbolic substitute tests. 

1. The classification of job activities proposed for the development of criterion referenced JTPT is a 
viable method for ykucturing the activities of the maintenance technician. JTPT qan be developed foreach 
activity and each can be measufed in its own right. ^ , 

2. The activities ^are not mutually exclusive. They can be arranged in a four level hierachy of 
dependencies as follows: ^ . ^ 

(a) Check out, remove and replace, and soldering activities.* • • 

(b) Use ol^general aiid special tQst equipment. . - 
* (c) Align, adjust, a(id calibrate activities. <> 

(d)^ Troubleshooting. ^ • * ' 

To Illustrate, tlie 'troubleshooting activity may include all of the activities in the first three levels. 
These dependencies havf implication as to the order in whi4h tests^houljibe^administered. For example, it 
should be ascertained that an individual can use his test equipment before he is permitted to take a* 
troubleshooting test. - ^ ^ » 

* V .3; aW considering £mdwcA process, and time as to their appropriateness for scoring the results fpr 
each activit^;^easured, it de^cided that a test subject has not-reached criterion until he has produced a 
- complete; satisfactory proc/wc/. This is a ^0, no-go criterion, * . ^ 

4. Considering the complexity and diversity of'maintenance activities, it was decided that abflities 
required to perform maintenance cannot be realistically represented under the umbrella of a single score. A 
profile is recommended which displays precisely what maintenance activities an individual has 
demonstrated that he can dr cannot perform. This giv^s a much more meaningful picture of an jndividiKd*s 
maintenance abilities. < . » - . 
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ThQ above consideriiti<Jns support ;tlie foHowing statements*/ 



1. Synibolic substitute tcsU^ should j)e developed and tried out for each type of 4naintenaQce< 
activity. The experience gained 'from the in-depth study of the nature of maintenance tasks made daring the 
development of the iTPT indicated the'requirem^t for symbolic substitutes-for all maintcnlince activities. 

2. The effectiveness of thV, symbolic test for each type of activity should be deterniined on its own 
n\erits» against the crilerion referenced JTPt for the same activity, Upte^s it*can, be deihonstrated that a^ 
symbolksul^stitute test has a high empirical relationship to. its criterion JTPT, it should not be used as a* 
substitute. .- S ** " . \ . " 



J 3.*^ Symbolic substitutes may be successfully devefop^d for some maintenahce'*iicfiyitiQs, but not. 

necessarily for all. * - > . , - * / ; . . '-,\-^'>. ^ 

• • • . ' ' ' - A 

tests. ' ' . ' . — i _..^v 



The saive implications for ordering of /ejst.admift'islration t)^^^^^ tg JTFfr apBly to symbolic 



Discussion of the Nature -of Maintenance Tasks . ' » ' % ^ <; 

» ' , - ' 

la addition to the above conslSer^kions concerning the development of cmpiricalfy valid symbolic 
tests for maintehahce,^special consideration mlist^be given to' the nature of the njairitenance tasks. The 
maintenapce man's jofc is comprised of many t^pk^ relate<i pfimarily by the electrical and/or auechanical 
organization of the equipment which he is responsible for maintaining. The refationships established by the 
equipnieQt are -of several types: ' \ ^ ^ ^. 

1. Cause-effect between components. ' ^ ' * ■ * 

2. Physical proxihiity and accessibility. ..^^.^ % . / 

3. Similarilies of function, appearance, and types of togls needed for maintenaifcer ^ ^ * * 

4. Descriptions provided in equipment documentation. ^ ^ ^ 

For'eyei)o maintenance task, there is a requirement for the maintenance, man to organize his mainJine 
of approach to accomplishing Jhe task. In additioa to this *'main linC there are many auxiliaiylines^thRt 
conirlBute .to the main linef. For example, the ifiain line could be to remove, service, and replace a nliin 
drive'shaft. The 'auxiliaiy lines would.be collection of the proper tools, obtaining the*prop,er solvents aiTTd 
lubricants for servicing, identification of the.proper pages in documentation, locating the niain drive shaft, 
^peKonnlng actions 'nnd making discrimination to gain access to thc shafts, proper use of the tools to 
disassemble ^sliaft connections^ obtaining the appropriate equipment for moving the shaft if necessary, and 
obtaining appropriate .information to perform this task (i.e.,^ steps, following cross-references in 
documeAtJition if necessary, finding cross-referenced documentatipn* consideVing substitute lubricants if ti\e 
recontoended ories are unavailable, following disassembly insiryctions, following instructions, on use of 
special tools, interpreting ambiguous instructions^ etc.). ' ^ 

' These auxiliary lines each require many discriminations, responses^ and use of SQiiie^infonnation 
stored in the main tenance» man's memory. Every one of the discriminations involve sorting tluough 
irrjsleyant information, or cues, to select the appropriate ones. Every response involves the.sclcction«of tlic 
appropriate t)ne froni amon^iany possible ones.' In addition to these job-oriented activities, yiere are those 
of the environmejit; human, physical, and"orgahiz3tlional. Each of tliese introduces certain distractipns; 
talking with co-workers,^ answering their'xiuestions, being human, unlocking combination^locks, finding 
keys, answering calls, and attending meetings or other organization functions'. * ' . • 

, . * * * • • \ 

The picture obtained from this desciiption of the maintenance man*s activities, is one of great 

distraction as a function of discriminations and responses tliat are essentially unrelated except by the needs 
of the equipment for which he is responsible. ; " 

The example chosen is representative of all electrical and mechanical maintenance activities^ except 



that troubleshooting probably represeflts an even greater burden. Troubleshooting requires evfen piote^from 
stored iWmory; in a decisign-making sense, than procedural maintenance tasks. The distractions arc siniilar, 
if not greater^ but \t is even harder to remain organized' in trouWeshooting tasks in the environmen.t of 
distraction. It must/ be emphasized that most of the **distraction'* from tlie main line of en/ieavor is 



^'^ • ^ ^ ' .... ' ^> 



' " \ieoessary to accomplish the^task. Even the dispussions between people are often necessary in the larger 
picture of maintenance. They communicate information about location of components needed for 
maintenance, absence' of people who have keys or other necessary ingredients such as spare parts or special 



- - ^ Pievious Syinboltc Substitute Tests ^ . . . • 

Symbolic 'Substitutes for performance tells have' a long or * a very short history^ depending on the 
definition of symbolic substitutes. In the decade of tjie fifties there;' was a burst of research activity in 
testing maintenance and especially troubleshooting ability, pagne reviewed this^activity in his Annual 
Review of Psychology article on "problem' solving ^nd thinking^' in 1939. A$ Gagne said in this review, 'To 
i ^ summarize, troubleshooting of complex equipment typically consists of problem^solving which is sequential 
' ' an nature; thefe.is a sequence of hypotheses that must be testecf^in order to narrow progfessively the area in 
which 'the malfunction is, located." Experimenters in this periocj were using a certain type of paper arid 
pencfl substitute for troubleshooting in their research. They we^e using test materials which represented the 
"problem solving" aspect of troublediooting. This sipproach is based ori an asaimptipn that ai\ abstract 
* * * . logic, such as that described by Gagne in his revieWoT '^Rroblem solving," is the **key" to troubleshooting. 

If this were true, there would be a relatively high correlation betv^een su^jh tests and* troubleshooting on the 
real equipment. 

But attempts to develop symbolic substitute tests which concentrated entirely .pn the cognitive 
aspects of "problem solving" did not produce high correlations. For example, there were two studies of the 
effectiveness of the tab test; ene by Cjowderi Morrison, and Damaree (1954) and the-other by Evans and 
^ Smith (1953). Both itfdicated very lo^ correlations. Those studies, and'jjsimilar studies discussed in 
^ ^AFHRLTR- 74-57(1), reflect the same paueti^of low porrelatiohV.None of t^^se tests .included any of the • 
.. • • "distractions" from the main Wne ofj:pro{)lem:5o!ving" found in the ml wotld of troubleshooting. In the. 

job environment. an individu^.musty f5rexampte.^;seyjp and operate ftis test equipment to obtain test poin^t 
' - information, .as well as, <>Biaift;instQictfons anUtnfori^tion from hi^ Technical Orders. .Thes^ are very 

essentia joB skflls:'^ut:^elr;;pe^rforT^^^ or breaks m the test subjects 

"problem solviTigy thofi^ pr^oceSes.i? Ii^does nol eper^tsii^i^^g^jpip^filPrope^ he also introduces 
false informatrdnT^(f fhese thought pTo^essis. In the symbolic te>§^^u(^^ the tab test, he is given this ^ 
information without the distraction of Wlhg his test equipment. As a7'e^/suc^'s3^ tests measure 
neithec,the abflity to apply a troublesTOOting strategy in a realistic joMike settftg^on^the ability to 
* ' perform the essential,^but distracting, support stills.^ '''"'•"•rv - 

. ^ Rationale for Symbolic Substitute Development ^ , ' f 

f • . , 

An hypothesis mdfrfyin'g the symbolic substitute tests, then, is that the process of maintenance is' ' 
essentially a ''hodgepodge" of activities. This picture of the maintenance process would have as the good 
maintenance man one who systematically organizes essentially independent and disconnected activities aad 
perceptions into his pfitformance. This is not an "idealized" picture .of the maintenance man. Previous ^ 
fesearch has treated troubleshooting as a logical, problem-solWng situation. The present writers regard this 
as only one element in the actual troubleshooting process. Other elements mclude specific bits of practical^ ^ 
* information, ways of finding specifics in documentation, having certain skills with tools and test equipmept ' 

and. so on. The |ood ' ' " ^ ^^-^^t. n u:*^ ^c-.^c *: ««j ^^^w.«i 

then), wjtK informatir 
he is called .on to 

?ubj^ct'$ job on the test is to "thread his way" through this "hodgepodge" in the sanfe way hd^n^Sl. In the V y 
realworld. . -* • * ' /^/y ^' V>r • 

This makes the symbohVsuljstitute test a "hodgepodge^ itsejfijj. ip not desigft^il-to^neasuje any * 
V "pure** elements such as logic or knowfedge of theory. It is designed to'simCiJate ' ** - — 

referenced peH'ormance. This makes ^tests very specific' to the petfofmance 
generality for accuracy of predictiwi. The techniques for genjerating symbolic substitut 

•generality, but not the tests generated by these techniques. Whether it gains predictive \ ^ . ,y - * ^ , 
' the correlation obtained between this type of test and criterion referenced perforraahcel ' X ^ . "^^J^^V 

This hypothesis underiyj^g^ the symbolic substitute test, that the maintenance process involves an 
' ' assortriient of discriminations and 'activities which are unrelated except that they are required to accomplish 

;/• ' * * . 11 > . ^ - . 

ERIC .'^ . ^.f 14 . 





*' • ' ' ' ^ '5 .1 . ^ I . . '<^.th. , . 

y ^ the specific goals of the maintenaiice task at harid^ diffm Jrom the hypotheses uis^d far many other types, 
" of paper«and pencil tests, A part'of our hypothesis istthal the successful pei^ormqnce of a maintemnce 
^ activity fiquires au)"organizing factor'' from the perfprtrier.'^and^u failure! of[the perfdrmerto vpnectly 
' ^ '[ organize the unrjelatf^d^crimimti'ons and action^ wUl^^^svilt inJU^.bj^Jng^Umdle to perforrn]the reqi^ir^ 
^ maintenance activity, • ;>* ' ' / f/^^f' ' '\ ^ 

^ A xiqroUary toMhisiis that the omissiSn of any of, these uhrelate/i d^criminaiipnsor responses^ wiil' 

• * reduce the C^rrelatitin'nf th^^ .fivmhnlir/^th^tiMtP tH^ts 'uifth nyvnv^Af.trt ^ k'iftfow4^**''^Ml^*^^ 

perfofhiancei 

' ^irst. 
[ ^ -him some 

/ instancer.i 
,1 "piecemeal' 

^ ^ predict6r ok avi.uai pwivrimajiu^ ca ui^ /wnuic icai. ,iuc tuiitiusiy^i^Hgucnveu iroin me premises ^Dove. 

The piecemeal test would not require th6 subject to ^*6rganize'* hi^ ui)M^4d dfecriminatipnsVnd re^ponses^. 

while the whole test would. An exapple of ;*piecpmeal** t^it i!em^ >/Ould be multjple-choic< tests, in* 
, true-falje tests on specific elements ot thft'maintenahce proa^ • ' 

. Anqther deduction from this ^jemise is .that a test whipK' abstracts jRfom the.'actudl'joh QnTy the, - 

; . "lo^c" of troubleshooting Without* including all the .ii^ceWw^^^ 

; setting tesi instnj.ments,*acces5ing documentation, eic^fJ^Si n^t^^^^^ perfofmance as / 

>Ula test which does include necessary dfstractions'of^t?^ 

- hy|)6ll)esi.s on wKich sypboli^ wbstitl/tes is 1>^M.'^''M4'^^^ largely-^/ 
^ function of th^ ability to engage 'in unrelated jdisbrkinahpn^iaii^l^tivjUeV^ same time to remain 

^ organized, incite of this iietbssary*: ^' " w ;:.vv^v -.^.^,./-v . . , . 

' ^ This hypothesis is .a'basic one 
"which has not prevjou^y been 
confirmation is' that the symbon 
this proves t^ be- 'true, the next 
would be ( 
tests that 

\ ' ■ confinrfation or dejiiai^of the{bi}?ic.^^tiiesis.^^ .be modifiet orstan^in iu , 

p/V^^ "^^^f^^^^^j^V^^ *e t:onlext of unrelateidatail and theneed to rernaMtr^ 

' ^ S .^^Bapiz?^#*oward l^^ 

' ^t>f ^fpjer^d pendiT/testji Wttl>,r^spjCj>|U^ symbolic subs^ituMests 

/ represent, the first s^eo iri the' direction V^f-testmb^m hv^vnthp^R.'ln ftik^rPciMtifta #0**^ 




■ Ukewi! individuals can be found whi^iia^S s^nlVVeral years oh nothing else but thjs one system.The 
pSTftatthis serie^of tests i^not:>(;e^t to b^>^^^^^ 

•Uie AFSC 32^X^ ted;ili(ilair.;Ohe;^fe|tf in'- this jifei^of .responsibility, was selected a? a vehifje for the 
investigation of perfpt^mance, testis^ v-/-'^ " •.. ' . 

This effort to ' (feyiWntboUc'substitites;]o^ the perforpjahce tests proceeded without refining or 
othewise tr«iting W'^d^aT criterion /eferehcW JJP: Thfe &ci«=h^ been to exannne and tn^ ou 

• techniques for sijJja^ih^ ^e pe/ortnah.^^^^^^ The JTPT areused as the cntenon for 

measuiighow welJ^e ^mbolirtejits the^^e criterionjelated factors. The measure of success of 

how WeU-ttiesiitiuiation wa^^iiccomplished is the to.which any given individual perfonns the same on 

. thitwo veBiogS of the testis Thu$.-^>hether theifldividual is high or low in proficiency should be equally 

■ de|pnstrited on either^ V perform;mce br^i^bolic Versions of the tests if the symbolic substitute tests 
•are relfable.'valid predictors. '-/'..', q ' , ' .' 

the development of the symbolic-substitute tests generally reflect tfte hypotheses and ground rules' 
discussed, in Section I. As itientioned«aaier, in the development of the symboUc substitute tests there was a 
deviatioMrom the basic hypothj^ of criterion referenced JTPT. Of the discriminations and perceptions, 
that would be required to bring ap' eaufpment bacjcjij.flight-ready status, only a representative saniple Was 
•takeil foruse iri^iT 8^mbofie>b¥itute test?; A htiij^^^ 

terf equSpjneirti'aBii loeating^plaaement parts;h.ayel^^^ administrative efficiency of Ihe 

^iisf:nii^tih$K^^ indudes- most Of the total task and is largp enough.that the total context is 
'^^^r.^\<.^xJ^tek^.fai tlie siibfect the impression that -this is a complete task. Certain other 




'SymbbIic:T(^ti&evdopineiit 
2^ f ^^sistembwkoirt! System checkoutof the AN/At>N-147 system requires thafthe equiprneht be set 
ap-on a beMch which has a special test harness instaUed. that provides for signal i^pirt and readout of the 
wsociated'xfispiiys: When a ladalf s<it is suspected of malfunctioning, it is removed from the aircraft and 
■ iiroU^t into the sh6p'where this checkout is performed. Further, this initial check serves to pomt to the 
. ' ' ■ Igeneral. nature ofthe malfunction-, providing guidance as to the locationof.the probl^n. In^the cntenon 
" — JTPT tKe teduuciai is provided with a radar set that has been misadjusted.tO;give a faulty readmg. He then 

• -/ must setup and run the checkout according to Technical Order (TG) procedures and determtae whether it 
- v/'-- isfunctiphing properly. " *^ - .'v 4 , 

d - ' Analyzing the elemertts of this tsMc showed that the miun actions lequked w'ere to make the proper 
•V connsctfons betweep the radar set compAnts and thelfest harness, to set the controls on the radar and on 
the' jtest. harness to the proper positi,ons,^3ll to follow an established set of instructions for checking th« 
' rada**^cldng capability. These a"ctions were require^ before the technician could establish that tlfc set 
w^infactifotfonctioningp^^ • - ' } 

/to the jPtPT, ills only neces^ry to grade his conclusion about the equi^^^ 
■ ^ of tiSipse actions. Unless he had performed them p^operiy and could describe the resultant symptoms, he 
'.- • -lUi'tfot cOjprecflyperfon^^ . • > . 

- • •- f ■ ta'simulate-fhis'tasK it was necessary to insure that the|yfli6oBc tadc required all of 

■ tifcs-acffons that,the real 'taSk didj apd no Vfor example, only the concIusiBiis or only the control settings. 
FinAer; the ^rtbolic cB^ays, would be non-responshre and yiddnovariaWe signal to the technician for 

• 'Int^rpietation. TRerefore, it Would be. necessaiyUto test this portion of the task differenfly. The important 
" . ' paft.of the hookup portion of the ^ is to ccSnMt the correct outlets of the radar. and flie test harness to 

^ iKe simuhtor that is:use.d..The symbolic approach taken to this action was to pjesent a display of the 
.-'-V-yaribus equipments jtequtfedi, plus^eral irrelevant items and request the technician to draw lines between 
! J- ihe pr6per,qu¥ets indicatihg the-re^uired hookup. The control settings were tested by having the techrajcim^ 
•C'^ -indicate -on Ihe display -wharthe proR^ position would l?e for each relevant control. The cjicckout 
V V'.'piocedures were sbnilaily checked s&iceidiey con^t- of a sequence of control poation settings. 



The rafional for selecting this particular method of symboliJuiy testing the techniciao*s ability to . 

* perform the^^iscriminatibns attd^- actions involved in the' checkout^ task lies in the analysis of the 
cue-ie^onse4e<iuirements of the real' tadc. To perform the hookup portion of the task the technician, jn 
the real world; is .presented with the test ^liarness, the r^dar set, and access lo a supply of assorted cable 

^connections. He must make discriminations regarding the proper connections,, either from his knowledge 
and i)ast experience or from his documentation, secure the proper cables, and physically make the hookups. 
The symbolic task, simulates the same stimulus fidd to which the subject, must make very simflar 
discriminations and^ responses. He is presented with an arriy of equipment (Rgure 1); some relevant, some 
not. I^rom this ariayhe must select^that jvhich is required and by drawing lines, indicafe the proper 
connections that ^ust be made. Thesfe connections, are indicated not just from one piece of equipment to 
another but from the proper terminals on each item of equipment. Next, the technician must indicate on 
pictorial display! of ^ch selected item,pf equipment the prpper control settings for'aecomplishing the 

"checkqut. ^ . ''f " - . " ^ ^ 

The final part of the checkout procedure, interpretation of a dynamic display (the- groundspeed 
indicafor must "lock-on" oto'^ pre-set input speed from the simulator within 20 seconds), could not be ' 
simulate^ realistically in the paper and pencil. medlumfConsequenUy, this discrimination was not part of 
the sample of di§criniinations. that, the symbolic substitute test takes from the total field required iii 
performance. Whether or not this necessary omisfion from the sample is important for prediction, will be ' 
answerfed by the nature of the correjation between this symbdljc test and actual performance. 

Tjliere are two system checko«s that are required by this sy^stem, one for the AN/APN-147 Radar and 
the other fqr the AN/ASN-35 Co^jSputer which provided the basic inputs to the radar system. The same 
approach has been taken to both checkouts. • 

Soldering. The development of methods for evaluating an individual's physical dexterity to perform! » 
task isjthe most difficult challenge for symbolic testing. It may be that such tasks as soldering gre' not 
amemm to testing by symbolic substitution. But if this is the case, it is not a serious problem to test this 
smafl, but crucial, task by pe;formance pnly. However, rather than prejudge the-outcome, a symbolic 
substitufe test was prepared on this task, .. 

TTie symbolic form of testing this skill must; concentrate on knowledges and representations of 
, physical acts, as all paper an^ pencfl tests do. The symbolic tests that were developed4n this area deal with 
the indfartdual's knowledge of how to proceed with the tasks. Whfle this is a necesjary condition, it is not a 

• sufficierit one. The subject stfll must .have the physical dexterity to employ the tools properly. This 
problem is sufficiently pronounced in this area that there is no assumption, possible, for example, that 
successful completion of the symbolic tests on soldering is a sufficient demonstration of the individual's 
abflity to solder an dectrical connection. The greater the physical skill required to perfohn the task, the less 
ralid Che symbolic indicator of that skill. Soldering represents the most complex physical sk3l required of 
the el^tmnic technician; and therefore, symbolic tests dealing, with the measurement of this skUl are the 
trust limited. 



the other hand tools that the tecftfiician uses are primarily small screwdrivers, wrenches, etc. These ' 
are les| demanding in terras of physical dexterity.and so the tests relating to tasks that require their use are 
less lunited by this difficulty. The least* demanding physidd skills of the electronic technician are those'; 
relatmi to setting equipment controls and making connections; These skills can easily be assumedjto.be part- 
of tlie formal repertoire of behaviors found in the population that are selected for entry.into m^ntenance. - 
specialbes in the United States Armed Forces, -*- ' ' 

• ■ a rf\F^' task.was divided ij^to two tests, diie to the number of related factors that ' 

• mfluerice the ioyerall soldering task. TTie bomiflete. soldering taik requircj^proper identification of ' 

• compcAwnt location, which may involve schematic drawing interpretation, selection of the correct 
teplac^i^nt cromponent, proper removal of'the old component,preparation ofthe new one forinstaDation, 
and flrpy, use of proper soldering techniques so that a good connection is made and adjacerlt comiJbnents 
are iK^damaged in the process. Whfle soldering .was the main ^bcus^of the testing,:these!>asMciated " 

•• requirdments could not be Ignored. r - ' • ^' '"'T^"'^" 

|liey&5f soldering JTPT dealt with just soldering. The technician was givenWal component and- 
~roId wMreto install -them on k printed circuit board. He was then graded onhowweD he solders them in 
^ place,|)ased upon inspecflon of the soldering job and comparfson to a" photographic standard. In the... 
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symbolic substitute for this test the .technician is shown a series of solder joints and must match^them with 
the list of descriptors that describe eadi (Figure 2)^ The difference between abOity to recognize a proper' 
solderjoint and ability to execute one is fully appret:iated. ^ 

The second soldering JTPT included all of the ancOlaiy components previously mentioned. The 
technician was given a module with a number of components on it along with a schematic of the modide^ 
His instructions were to replace several components identified on the scheifiatic. He then Had to translate 
from the schematic to the proper componen'feton the module, select the correct replacement parts from a 
supply of assorted parts, and property install ttfe new component. In the symbolic version of this test, he is 
provided the same instructions and schematic; however,jnst0ad of the actual modtde he is given a series of 
^physical drawings (not schematics) of the module (Figure 3). He is to^lect the drawing which depicts th? 
proper execution of the instructions. The technician must translate from the schematic to the actual 
module in the same manner as in the performance. test and must be aware of such factors as polarity and 
coding to select the proper drawing. .Again, however, the actual «kill in executing the soldering .task is not 
represented. . , ^ 

, Remove and Replace Tests, The other tasks of tl?e electronit technician that require use ofTikhd 
tools primarily revolve around removal and replacement of modules wfilnirthe radar and computer systems. 
The prime requirements of these tasks are that the technician be able to correctly locate the module he 
must reihove and accomplish the removal and replacement without damage to the equipment. For 
performance testihg purposes, the technician was required tq remove and replace a series of modules, and 
-<'was graded, upon proper selection, successful removal, and replacement. The latter was determined, by 
whether th6 equipment functioned properly upon completion of the task. For" the symbolic tests, detafled 
photographs *of the modules were providfe^ to* the technician. He was nequir^d to select the proper 
photographs based upon the^ddule that he was tcJji to replace, and then mark the fasteners that woul<ft^ 
have to be disconnected in order to release or gain access to Aesubjectmodule. • " . * * ^ 

General Test Equipment Usage Te§ts. Another, general category of activities required of the AFS£ 
328X4 technician is the use of standard electronic* test equipnient. This equipment would be used in 
virtually all types pfelectronic mainten^ce. Those general items of test equipment required for ugg,wi^ 
the AN/APN-147 system are the PSM-6 Voltmeter, the 545-8 Tektronic Oscflloscope, the 1890M Transistor 
,5 Tester, and the tV-2 Tube Tester. Whil^in practice any nupiber of other types of simflar equipment may 
be substituted for these specific items, thos^e that are specified by the official maintenance dc^umentation 
were used as the standard for testing purposes. - 

Voltmeter Symbolic Test Analyses of cue-responses presented by these items indicated that they 
presented a fairly wide-range of requirements that had to be treated individually. Tjiese items of test 
equipment ranged both in mode >of application an^ coniplexity of usage so that no single symbolic testing 
strate^ ^oiild be applied to all. The voltmeter is the most widely used and the most simple to operate and 
interpret Typically the technician is given a V0||tage value that should be present at a given pointin the 
, system by his documentation. He then employs the voltmeter to determine^ whether that voltage jsjin fact 
.present at the specified point Thus, he is coMparirig actual valiies^with giyen ones! In the JWT, the 
technician was given a series^ of values that wire supposed to be present at given points oi^a voltagp 
simulator. He took readinp at the specified points and indjcated wjiether each was within accept^^le limiti 
of the specified values. f"" 

For the symbolic version of this test, he was ^Ven^the same specified values (Figure 4)|and then 
shown a voltmeter display for each problem (Figure ^.|He had to interpret the display against the given 
vaTue and make the same determination of whether or not it was withm tderancc limitSi An alternative 
strategy for this test was tried out, in which the technician would take in unmarked dispfay and indicate 
the control settings and needle position necessary to give the specified values. While this was more realistic 
in terms of the setting of the controls, the drawto^inof the needle position did not correspond to any job 
activity. Further, variations and inaccuracies introduced by having the technician draw in the needle 
pofition made interpretation of the results difficult. The use of the pre-set (display which the individual had 
to interpret was felt to be sufficient exercise of whether thi technician <fould pfoper^y set the controls 
himself,sincethereareonly two involved on the voltmeter. *^ * / • ^ ^ * : 
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OscUloscope Symbolic Test The use of , the oscilloscope follows the general pattern of the voltmeter 
except that it is considerably more comfilex. Again, values are specified in the TO for signals that are 
supposed to be present at given points. TllliSoscilloscope is employed to verify whether such signals do in 
» fact exist. There are, however, many more cwltrols on the oscilloscope than on the voltmeter increasing the 
. number of discriminations and responses required to perform adequately. The os^lloscope also displays its 
results as wavefq^ms rather than meter readings, which means a more complex type of discrimination is 
required. However, the same basic format is use*d as inhhe voltmeter test, even though the number of 
discnminations and responses is greater. A waveform is given to the technician along with the necessary 
extWanatory information. In the JTPT, the technician takes readings from specified points on a waveforrii 
generator and compares the resulfs with the standard. 

For the symbolic test, fht technician is given a pictorial display of an oscilloscope, as shown in 
Figure ^.'^He must interpret the display (i.e., the interaction of the control settings, wavefonji, and grid 
notaMons) and determine whether the simulated output is in tolerance with the standard given in ^e 
problem. The technician is given seven problems that exercise all of the features of the oscilloscope from 
test probe calibration to dual trace. Because of the. multitude of controls on die oscilloscope, however, it is 
not as likely that correct interpretation of the results on any particular one of these seven problems implies 
abflity to property set all of the controls, as is the case in the voltmeter test. Therefore, in addition to 
results interpretation, the subject must indicate" the correct position of certain key controls for each 
problem. This insures that he has properlv^4 the ^ope for determining the answer. The same technique is 
used1n\.both the performance and symb<aic versionslof the tests. In the performance version, die subject 
" must obtain an Actual waveform on the oscflloscote, which requires setting the appropriate controls, 
whereas, in the symbolic version the display is prodded to him and he must indicate the appropriate 
settings with a penciL 

The specific technique of having die subject mark die position of all of the cofttrols to be used for a . 
given function and draw in die resulting waveform on die oscilloscope was considered and rejected for 
several reasons. ^-'-^^^^-.^w-. - • 

* First, the same objection as in die voltmeter test was encountered; that is, diat drawing a wavefoijn 

on die scope face has no counterpart activity in Uie real >yorld. SecoM, while indicating the position of a 
given control wiUi an arrow or mark vtqnld correspond rather well wiUi actually setting it, there is a critical 
difference between die actual, and synfbolic task on the oscilloscope display. In actually using a scope, die 
technician can employ trial and error on die controls to try td>,achieve what }ie is after. This serves to 
jC^frfifdi )iis memoiy and to provide a very useful form of feedback that helps him perform the task. As a 
' gross example, if the technician forgot to activate Uie POWER'ON switch in actualperformance, he would 
not proceed far before realizing what was wrong. If he overlooked it in die symbolic test, however, he 
would get no such feedback. Thus, this type of symbolic test would be over-demanding of the individual's 
knowledge of die oscflloscope. For diis reason, an approach was taken diat also offered die individual some 
clues - not the same ones as in die actual job, but analogous ones - by naming a small subset of controls 
and focusing his attention on die settings required for each of them. : 

Symbolic Tests for Transistor and Tube Testers, The odier^two general test equipment items "Se 
simflar to each other in function and both different from die voltmeter or oscUloscope. The purpose of the 
transistor and tube testers is to determine if a suspected part is good or bad. The 1890M transistor tester is 
deagned to make incircuit xiiecks of transistors. The JTPT are designed to determine wh^dier the 
technician can distinguish between good and bad transistors when using the device. When translated to die 
Symbolic medium, the task becomes one of uang the 1890M instruction booklet to set up the device 
according to the given specifications of die transistor to be t^ted. This is done by die technician looking up 
the transistor value and accompanying device setting instructions, -indicating them on an Answer Sheet 
display of die device (Figure 7), and drawing in the Jimits of acceptable values that would ir^dicate a good 
transistor. The TV'2 Tube Tester follows exactly the same format, providing a technician With a display of 
the device (Figure 8) and die accompanying instmotions that he would have in the job en^ronment, 

^specifying a tube to be checked, and dien requiring him to indicate die^proper device" control settings and 
the acceptable range of readings for a good tube of diat type. 

Special Test Equipment Usage Tests. In addition to the general type of .test equipment described 
previoudy, the AFSC 328X4 technician must also be able^lo employ several specialized items of test 
equipment diat are not common to many other electronic maintenance tasks. These arerfhe AN/URNW5 
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^ . Figure 6. Oscilloscope disphy. 
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Signal Generator, the TS-382 Audio Oscillator, and the CMA-546 Doppler Generator. These items of ( ^ 
equipment are specificalty related to particular functions on the ^^N/APN- 147 Radar and,' consequently, 
must be tested in that conte)»t. The demonstrati6n of the subject'il capability to use these devices, then, is • 
done in the context of specific Maintenance tasks on the radar set. The difference between these-tests and . 
those in othe^ cat^gori^JS that also require the us6 of this equipment is iiv what is graded. In these tests, the 
individual's ability jlo properly set up and employ the items of special test equipment is evaluated in.a ' 
manner similar to* the genera! equipment tests. In the adjustment and checkout tests, where these items are 
used as part of the distractive aspects of a larger task, the evaluation is based upori accomplishment of that 
larger task, not the distractive discriminations and responses that are auxiliary to it. \ ' 

ih To demonstrate that he can properly employ this special test equipment, the technician is presented ^ 
a proMem that requires it^ use. He then must indicate on a suitable display, how this item of equipments 
'connected to other units and the control settings*^ that would be made on the subject piece of test 
^equipment for a specified check. Figures 9 through 13 show the displays for the special equiprh^nt tests 
which the technician uses to Indicate the control settings. - } • ^ \ ' 

Dependency Relati§nships. The foregoing' series of iests - checkout, special arid-, geneir^ test 
equipment,-and soldering - represent discrete, elements of the larger tasks that are the primairy funct^n of 
the electronic technician. These larger tasks are those of equipment repair and maintenance cofrip<»ed of 
a^justirig and troubleshootirig the equipment. T^iese- tasks are 'also tested via the perf6nnari(;e\tests,, 
howevef, the individual components of these tasks are not graded separately. Thus, while these JTl^ w^J 
indicate whether or not a technician is fulljr qualified on the maintenance of the AN/APN-147 system';lhf^ , 
will not yield diagnostic information about speci% weaknesses without the subtests that ded'wiijjf tfie^- 
individual elements. In this way, the tests of the total maintenance tasks are not graded'in sucltamanrW^^a^ * 
Jo cQpfound the , evaluation process. Troubleshooting, for example, is not graded on the,-?jasis^-<?f\ille.u ; 
particulai: procedures fdlowed or some composite of the number of enors and assists the indlividi^-m^^ - 
and received while solving the* problem. It is simply a matter of whether or not the su^ecf was-aJ)1^G^' ; . 
satisfactorily repair the'equippient and bring it back to'flight^ready condition, sy mboli^ly w «^(ual^^t^^^ 
purposes of simple evaluation, the troubleshooting, and adjustment tests ^ould'be si^^^i^Fr^rng^^ 
detailed analysis of performance strengths aiid.weaJaiesses, however, the other tes^J/ould,i)^;requu^ > . , " 
Also, the hy4)othesis underiying these tests is that air of this "auxiliaqr j^r^^nra^J^ 
•distraction* in creating the job context." „ . --^'X<^.^^.- ' ^iiO 



AligAment Tests: The process of bringing various })arameters of elecff^<j^ui^iti^irUo^.on>cr 
performance tolerances, is variously referred to as alignment, adjustment^ of ^ibwti^lh;ih|^^ ^ v 
series of JTPT, there were both adjustment and alignihent series of tests kfid^M^^ 
' two was simply a matter of how they were refened to in the TO. No dgritfi^^^^nfl^ 
definitions or distinctions have been developed or areln^existence to the J^^ovdeof^^xsfil^^i^^ 
•1:1. strategies for testing these ta|cs were the same in the performance tests and are necessarily^ " 
symbolic versions of these te^s. Therefore, the distinction will not h^ retained " 
the term alignment wiQ be us^d to encompass all three terms. 

• In the performance versiori of the alignment test^ (there are a total _ ^ . . 

JTPT) a given parameter of the radar or computer system is placed out of alignment by a knoyit, standVd«^ 
amount The technician is" then told to perform a certain adjustment ' ' ' " ' *^ ..-...r..^^^^. 
set being brought back into performance tolerances. If at the ( 
demonstrate on the appropriate item 6f test equipment that the set 

the test (The technician is told what adjustment procedure to perforin rather than having to decide wjial^tp ♦ J^IM 
do in the interest of saving test administration time. If the adjust problc^nis were treated as troublpshpotiiq;. ^j, 'j^^ 
problems (faulted parts), the" technician would have to spend an unacceptable amount of testing fim^^"";r^rv^'j 
checking out a wide range of possible problems.) Thus, the troubleshooting and alignment aspects of the^, ^-^ 
job'are'kept separate in the tests. . ^ . * ^ " 

\ To test the technician's abflity to make the same alignrnent in the symbolic version of^tfieJe^^it is 
necessary to shift from the desirable objective aiterion oT"is the equipment back in adjustment^' to a^niofe. 
procedurally oriented criterion, this. is necessary §ince there is no feedback from a •live" sygtem^to 
determine successful completion of the activity. The'symbolic substitute test provides a display; of a wide , ^ 

rangi? of test equipment,^ from which the technician selects and indicates those that Would be used and T ^.^^ v 
draws lines to properly connect them for use as in the checkout bsts."(Eigu"res 14 ^ 15), For each {ucce of . 
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Figure 15. Radarset checkout answer sheet/ 
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equipment used, he must indicate on a picture of that equipment what the proper control settings would be 
as weU as what indications he would get which would indicate that the set was operating correctly. Further 
he has to indicate the particular control (trimpot, etc.) on thf set that would Re used to njake the required 
adjustment There is still a 1 ot of "distraction" in the test^ 

Due to the need to keep the amount of symbolic testjng hiaterials withi;i manageable bounds for 
these tests, it was necessary to limit the acceptable approaches tocjtiaking the test adjustment to those that 
are sanctioned by the TO. hi the JTPT, it was possible to permit the technician to use any procedures and 
equipment that he chose. All that was graded was whether or not he successfully accomplished the reqfuired 
adjustment For sym&o/iciesting, there is no indication from the real equipment to show whether it is 
night-ready or not Th^ore, it is necessary to grade on the conectness of the procedures nsed. The 
grading standards that are imposed are based upop the methgd specified by the TO for making each 
.adjustment One of the prime characteristics of the symbolic approach is that it necessanlyjimits the range 
of choke to .that which has been provided. It did occur in administering the performance tests that 
equipment required by the TO for a particular task was not available in many shops. Such discrepancies 
between prescribed and actual procedures complicate symbolic testing. One adyantagf of the symbolic 
adjustment tests over the performance tests, however, is. that they^do provide a coiivenfent "audit trail" of 
an individual's performance.. Analysis of the detailed answer sheets can provide diagnostic information that 
is not captured by the performance tests. « 

troubleshooting Tests. The troubleshooting process has been of continuing interest to researchers 
for many years, and there have been many studies directed at -describing, teaching, and improving the 
processes involved. The purpose in this study was to simulate it rather than explain it. In observing the 
actions of technicians when troubleshooting, it was. found that the infomaUon sou^t for any given 
problem varied considerably. Some based their inforitiation requests Pn a faWy obvious mterpretation of 
the malfunction symptoms observed. Others appeared to be operating largely oh less obvious expenence 
factors and relating the symptoms to previously encountered difficulties, while still others followed eveA, 
less' decipherable 'patterns. Testing such random patterns of trouble^ooting, without penalizing Ae 
individual for using the wrong "process" while solving the prdWem, was accomplished in the performance 
versions of the troublesljooting tests. The technician was graded solely on whether he was able to render a 
set which was inoperaMe due to a known, planted $ult, operable. Whei; translating to the symbolic 
mediun/*e initial problem was encountered as in the adjustment ^ea; tfie. non-responsiveness of the paper 
simulations provided no feedback as to correct accomplishment of th6 task. While (Jiis could be handled in 
^ the alignment area by specifying the process, this was not possible in ,the troubleshooting area due tojie 
lack of an absolute "right way" of solving each problem. It was necessary to provide some semblance of the 
high de^ee of latitude allowed in the actual troubleshooting process if the tests were to be accurate 
reflections of Jjhe job proceij. . i . 

It was decided that the test had to provide a store of information that the test subjeoT^ould tap at 
win, much as the actual equipment contains all of the information the test subject nee.^s to solve the ' 
problem if he can ask the right questions and conectly interpret the answers. Preliminary approaches were 
• tried that gave the test subject infoh^tion based upon a theoretical "branching" or/'action tree** structure, 
The difficulty v;as, as mentioned previously, that few technicians used the same "tree." What may make 
sense to one, would be unenli^tening to others. Further, there.was th? problern^of giving away ^ good bit 
of information by having to confirm or deny thf accuracy of ^ch step. ' ^ 

As this approach broke down, it gave way to the idea of simply provi<fing the Information to the test 
subject in a "cafeteria"* style, letting him choose any information he desired and make his decisions and 
conclusions on his own. This has a great deal of simflarfty to the actud^ froubleshpoting process, but the 
probleifts in executing the idea appeared formidable. Trials with.giving^"good-ba'd" indications, or with 
providing printed voltage read-outs .were not satisfactory in thacf these were not the type of displays the test 
subject received from the equipment. It was finally decided to provide the display of the requested 
information as the test subject would actually see it. If he wanted to know tiie signal/present at^a given 
point, he would be shown a picture of a voltmeter, .osciOoscope, or otiier appropriate item of. test 
equipment with the requested information depicted on it in the same manner as if he ^tually took, the 
reading. This provided Ixtensive "distraction" with reject to the "main line**, of troubleshooting, but not 
as much as i(i the reat situation. 



' I 

* The next problem in accomplishing this approach was to anticipate all of the possible requests and to 
.document what the actuaK readings woufd be in a set with the specified fault present. While it was 
physically impossibly to document and provide to the test subject a complete set of parameter values for 
every possible condition, a sufficfent populatioh of values (that would be requested by a test subject who 
had some competence) was developed. In addition, a standard set of answers were generated to be given by 
the test administrator to questions that dealt with values outside of the bounds of the problem at hand. 
When the test subject got too far afield, he was .told that the requested signal values were "in tolerance," 
rather than being shown the actual displays. WhSie this serves to limit the wrong paths the test subject can 
follow more than the actual criterion JTPT, it did not, in practice, provide significantly more useful clues. 

One anticipated problem was how to prevent the test subject from **playing the test" by simply 
requesting an abundance of information until he got an indicated answer. This certainly was one of the 
drawbacks of the **good-bad" type of indications previously considered. Beyond converting the feedback 
information to actual displays that had to be interpreted by the test subject, the use of time penalties fdr 
each piece of information sought was ccMisidered. In this, if he asked for an item of inform'ation that would 
have taken him 15 minutes to acquire in the real world, he would be penalized 15 minutes on the symbolic 
test. This would reduce the amount of random guessing that he might pursue. This was considered when the 
bank of information was being provided to the test subject for his own use. The scoring problems associated 
with the administration of this system, however, were undesirable. Instead, it was decided that the test 
administrator would serve as a filter for requests for information. The TAhas the manual that contains all 
of the system values for the problem under consideration. When the test.subject req\|csts information, the 
TA looks it up and shows him the requested display. The test subject observes it' and records what he 
considered necessary and continues on from there. This prevents random perusal of th^displays and also 
tends to encourage the test subject to ask more considered questions. The TA then responds to requests by 
either showing a Relevant display on a piece of test equipment or gives a standard response if the requested 
information is outside of the problem boun4aries. In either case, the response of the TA is "value free." He 
is not to give any indication as to the relevance or irrelevance of the question. This display function should, 
of course, be relegated to some neutral mechanical ^device/Development' of sudt a device, however, was 
beyond the scope of the present study. 

Summaiy Concerning Symbolic Tests Developed.; In, a previous effort reported in AFHRL-TR- 
74-57(11) of this series of documents, a battery of criterion 'ffeferenced JTPT° was developed. These tests 
were developed for the key activities of the maintenancejobs concerning electronic systems. The activities 
for which such tests were developed include checkout, soldering, remcml and replacement of components, 
use of generd and special test equipmer^, alignment, 'and troubleshoQt1n|. JTPT of4his type require actual 
prinie equipments and test equipments, for their administration and can be^^administered to only a limited 
number of subjects at any one ttae. These limitations make the administration of criterion referen9ed JTPT* 
^expensiye. In this' section, the development of a battery of symbolic substitutes for these JTPT is described. 
The objective of these symbolic substitute tests was to overcome the expensive administrative cost of the 
criterion referenced JTPT but to retain the empirteal validity of the JTPT* Attempts were made to simulate, 
as closely as possible, the actual tasks of the criterion referenced JTPT using paper and pencil simulation 
techniques. The symbolic tests developed require no actual equipment for th^ir adniinistration and most of 
them can be administered to a group of subjects. In order to accompli^ this administrativex>bjective, af 
least some of the -realism of the criterion referenced JTPT had to tie sacrificed. The remainder of this 
document is concerned with ^ow much these compromises affected the empirical validity of the various 
symbolic tests described. . ^ 



ni. SYMBOLIC TEST EMHRICAL VALIDATION 
TcstTVyoutPIam \ 

During the development and tryout of the symbolic tests the maintenance personnel of several bases 
Of the Military Airlift Command (MAC) and the^Tactical Air*Command (JA^ were involved. 

1. During the development of the initial symbolic: tests, several contacts were made with th*e 
maintenance personnel at McGuire ^FB, NJ. The contacts were for the purpose of obtaining information as 
to how the various maintenance activities were performed in MAC. ^ ' • 



2. After the original symbolic ^substitutes wre developed, they were by the 
contractor at Norton AFB, CA. The subjects were experienced personnel with the AFSC 328X4. This was a 
very limited tryout The purpose of this "shakedown" was to determine the administrative feasibility of 
each of the symbolic tests. At that time, no attempt was made to determine their empirical validity. It was^ 
found that all of flie tests could be administered without too much difficulty with the exception of the 
troubleiiooting tests. The presentation of pictorial test point information using a book media proved^ 
difficult in the symbolic troubleshooting tests. All of the symbolic tests were modified to some extent as a 
result of this "shakedown.** . 

3. The ori^al plan for ascertaining the empirical validity called for the participation of the MAC 
Maintenance Standardization Team. It was planned to have team memt^ers administer the criterion 
referenced JTPT and their matching symbolic tests to experienced electronic technicians. This was to be 
done during their official evaluation visits to MAC maintenancesquadrons. As explained eailier in Volume 
II, the sVa»dardization team did not make as m'any^isits to maintenance squadrons as ori^nally planned. 
And*d\xnng the visits that wete made, there was insufficient time available for administering the criterion 
referenced JTPT without regard io the symbolic t^st\ 

4. As far as the empirical validation of the symbolic substitute tests in MAC was concerned, it was 
necessary to use fourteen novice subjects with limited training. This validation was performed at Altus 
AFB, OJC This section concerns the vdidation at Altus AFB. 

5. Arrangements for experienced subjects were later made with TAG. The TAC validation took 
pla(Je at Langley AFB, VA, and at Uttle Rock AFB, AR. This validation was concerned with 
troubleshooting tests only. TJje modifications of the troubleshooting symbolic tests used for this validation 
are presented in Section IV of this document." The TAC validation plan and the results are presented in • 
Section V. ^4 i 

Description of VaWation'Ahus AFB ^ 

As stated above, the purpose of the validatio^effort at Altus AFB was to determine the degree of 
empirical validity of the symbolic tests for various Electronic maintenance activities including checko\it, 
soldering, use of test equipment, alignment/adjustment, and troubleshooting. In the Altus effort, flie 
symbpjie-testr^wiscompared against the JTPT as the criteria. In this comparison, a group of novice 
technicians were testeS^n a series of problems with botE the symbolic and performance forms of the tests. 
These test subjects iere part of another experimental project. They had just completed a special. Uaining^^ 
course in the maintenance^ of the equipment for which the teste were designed. As such,ithey represented 
convenient population available-for testing. . " ~ ' - 

Thft 14 test subjects were all 'novice personnel who had not bcen^ trained in any of the stand5i4 Air . 
Force electronics training courses. Instead, these personnel were selected directly from basic training aild 
given an experimental four-week training course in electronics mainteifance using newly developed Fully 
Proceduralized Job Performance Aids ^FPJF'As). Tliese FPJPAs were ^designed to guide maintenance 
personnel through the otganizational maintenance activities associated with the AN/APN-I47 and 
AN/ASN.35 system. (These FPJPAs' and the associated training hadvbeen developed and administered 
independently by a separate contractor. For a complete report concerning this effort,.see Mullen ai^fi Joyce 

0974)). ' ; . ^ '^ : ^ ; 

Once the test subjects had received their ti^ining, they entered the testing program. The primary 
purpose of the testing program was to determine tfie predictive validity of the symbolic tes^s, and not to 
evaluate the test subjects, the FPJPAs, or the trairiing. course, for this teason,,the test desigh employed is 
different tlian would typicafly b4* employed fo^ evaluating^ training or documentation programs. To gain 
maximum informatiotn on the symbolic tcsts^in the relatively limited time avaaahte for testing, the 48 
symbolic tests were divided evenly (timewise) among half jptf the sulqects, and each subject took fiyc or six 
pairs of tests- However, no test subject took more tiian o)ie pair of tests in 'a given categoqf.^Thc testing, 
schedule was then repeated for the second group of seven subjeits. / * ; / ^ - ^ , 

Testing wis conducted in a specially designed' trailer that, contained a complete bench testing and ^ 
checkout, set up for the doppler radar and computer, plus all of the required items; of test equipment. 
Subjects were tested at the rate .of two fix day and each siibject took thc.assigncd test twice; once as a 
JTPT and once in 'the symbolic formats The testing schedule wa^i^arranged^o that each subject took the 
symbolic version of the test first m half the tests and the perfprmanoe vcnitSi/Zrif for the other half. This . 



sequence was then reversed for the second group. Also, the tWo forms of any given test were not given 
sequentially, but dispersed in time. Typically, a subject would take several symbolic te^, then several 
different performance tests, then the performance version of the earlier symlrolic tests and TmaHV the 
symbolic version of the first pei^ormanSe tests. ' 

Except for the case of the troubleshooting fests, there was no single correct "answer** that could be 
learned in one or theiother version of the test. The test subjects had to be able to "perform" the tasks in 
both version^. There was a possibility of practice or learning occurring during the first test on a given 
. problem and the counterbalanced administration schedule was used to equalize this possible effect. It was 
necessary to give the same problems to an individual on both forms of the test O)erformance and symboUc) 
in order to establish the **pairs'' on which to base the correlation. 

Analysis of Results \ o 

The analysis applied to the results obtained was designed to yield information regarding each discrete 
category of test classifications. This was done because each test category represented a unique set of 
materials and testing strategies. The purpose of this study was lo find which tests showed promise so that 
decisions regarding further investigation of symbolic substitute testing could be made; 

» As a measure of relathre correlation between the two types of/test given in each category, the phi'' 
coefficient (<t>) was used (Hays, Guilford & Fruchtez, 1973), comput/ii from according to the formula: 



iri which = S 



(fo-fe)' 



M*ere fo = the number of similar results observed 

fe = the number of similar results expected 

< The data necessary for these computations was derived by casting the results of testing into a serifes of 
contmgency tables. The 2 x2 contingency table appUcable to each classification oftestsis shown in Tabl\l: 

/ Table, L Example of 2 x 2 Contingency 
Table for ^ Statistic 



Matched 



Miimatchtd 

n»«ttiti 



Expected ^ 
Observed 



a/2 

X 



i 



all 
n-x 



I 



That is, if there was no relationship between the Symbdic and performance test results, we would 
expect that in haff of the cases, the scores would be thet same on'both and in the other half they would 
differ, based on dhance alone. The direction of difference is not of importance but the Incidence of 
differences is impdftant. The tests were desfgned with the intent that if an individual fails a test item under 
one test condition, he should also fail it under the other. Simiferly, If he passes one fbrni, he should pass the 
other. , ' . - ■ 

^ ' Table 2 shows the results obtained in the tiyout by category of test. The raw score given in the " 
matched* row is the total number of paired administrations in which test subjects attained the same score 
( pa» or fail ) on both the symbolic and perfo^nce version of the tests. The "differ" row gives the 
nifmber of test pairs in which the scores achieve^^on- the two tests were different. The rekder is again 
reminded that the subjects tested were «o/ experienced electronic maiiltenance p»isonnel h\^t novices given 



a four -week training program on how to use test equipment, how to use hand tools, how to remove tiijd 
replace components, and ho\y to use FPJPA to perform checkout, troubleshooting and aljgn and adjust 
•activities. It should be noted that these novices did mt have FPJPA to guide*"' them op use of test 
equipment; hand tools, and how to remove and replace components. They were supposed to-be*able to 
perform these activities as the result, oniyf ot their training. In this respect, these subjects were similar to 
experienced personnel who are usually expected to perform such activities as a result of their on-the-job^ 
tramftig (OJT) and their experience. Column 5 of Table 2 indicates this difference in support of activities.* 
••♦JPA'* indicates that the subject used a FPJPA to help him perform tKe activity. "JOT" Indicates that he 
was expected to perform the task completely on his own as a result of job oriented training exercises. 

Table 2. Indicating Validation Results and Type of « " 

Training for Each Activity Measured 



Activity 
Tested 


N 


• 

Retullt 
Matchad 


RnuHs 
Differ 


Type of 

Trafnln^* 
and T«st 

Directions 


Checkout 


4 


4 ■ 


0 


jpa''. 


Soldering 


4 


2 • ■ 


2 


JOT' 


Remove and Replace 


14 


" 10 


" ' 4 


JOT 


General Xest Equip 


6 


5 


1 


JOT 


Special Test Equip 


6 


4 


^ Z 


JOT 


AJ ignme n t/ A dj ust me nt 


19 


15 


4 


JPA 


Troubleshooting ' 


9 


? 


6* 


JPA 


p _ 

. *A11 subjects received 4-\veeks of Jao Oriented Training on how to use test equipment, on how 
to solder, on how to remove and replace components and on how to use Job Performance Aids (JV^)* 



JPA indicates that tests on activity was performed with Job Pc^for^nancc Aids (JPA). 

^JOT indicates that subject had no JPA for activity and was expected to perform as a result of 
hisjob Oriented Traimng (OJT). ^ 

The contingency tables for each test categories ate given in Tables 3 through 9.^pplying the 
previoudy defined statistics to these data, the correlatjotts (shown in Table 10) were obtained. Due to the 
rather small N for each type of activity, no attempfnas been made tp estimate the statistical strength of 
each relationship - rather they should be conddered only as treiids. It will be notpd that with the 
exception of the tests for soldering and troubleshooting, all the correlations are. positive.^ Comments 
concerning the results obtained concerning the tests .for each type of activity appear in the following 
paragraphs. " ^, ^ ' * . 

Review of Symbolic Test Results by Categoiy . 

Checkout Tests (<P - 1-00). AD test pairs of criterion referenced JTPT and symbolic tests taken in 
this category of activities produced the saJne scores. The checkout psocedure was not anticipated to present 
any problems; however, two of the four subjects wi|re not able to conectly accomplish jhe procedures on 
either form of the test. The computer checkout routine used in the FPJPA was considerably longer man the 
one normally used in tjie field, and a tfrne^adjustment was made in the JTPT to allow for this. 



Table 3. Checkout (0 = 1.00) 



Table 4. Peripheral SkiOs 
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Tabte 9. Troubleshooting (♦ r 33) 
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Tt0IeJO.. Indicating VaHda^on Results. )^ and ♦ 



Cbeffident for Each Activity Measured;^ 



Activity 
THtad 



Mrs 



RMUitt 

Matchad 



RMutts 
» Olff«r , * 



Checkout 
' Soldering 
Remove arid Replace 
General. Test Equip 
Special test Equip • 
Mgnment/Adjlistment 
Troubleshooting^' 



4 
4 
14 

"6 
6 
19 
9 



4 

2-, 
10 
5 

' 4 
15- 
3 



0 5 

4 



4.00' 

•0' 
237- 



1 / 2.67 



,2 
.4 
6 



.67 
637 
1.00 



♦ 



.IXX) 
0 
.43 
j67 ' 
. 33 
'.5» 
, -.33 




•'$1 ;<^' 38 



^^^^ 



V 



Soldering ^0 - 01 As was anticipated, the soldering Jests results showed tittle correlations between the 
two test forws. The Physical ^equireiioenls are extremely essential to the determination of >^ether a man 
can. solder or not and this was demonstrated in PT-2 (circuit board soldering) where both subjects ppuld 
recognize a good solder joint, but 6nly one fould produce such a joint* In PT-2 (conventional soldeHng), 
the subject wh^ faSed both test forms was unable to identify replacement parts (resistors, capacitors, etc*). 
In ,the case of the individual who performed the actual replacement conrectly, when tested via symbolics, he 
did not deal with pdarity properly and selected the yrong answer- Hs correct performance on. the 
performance test may have been due to diance insofar as the matter of polarity was concerned* 

^ Jtemoval and Replacement (<t>^ - AS). The results in this category were somewhat higher than 
anticipated^It was generally expected that the vast difference in feedback available in these tasks would 
permit* success in the JTPT ttfat would not necessarily be reflected in symbdic tests. Two patterns were 
obsci^d. One was that subjects , generally failed because they could, not identify the described module. 
WienJ this occurred, thiey were lu^ucc^fuji in both versions of the test , since they were given no 
confumation of their actions, righf/or wrong, in the first administration of the test. The second feature of 
interest is that, of the four subjects who passed one test and-faSed the other» alLfailed the initial attempt 
and passed the second; thh was true v^ether taking the symbols or JTPT first. This suggested that without 
the learning effect of thelest-retest, ttfe first trial results wodld be the same. Thus, the correlation appears 
to have been reduced by learning efifect rather than test format ^ffexenct.JTie symbolic substitute format 
appears highly promising in this area in spite ofJhe imperfect correlation obtained.^ 

General test Equipment ^0 - .67), General test equipment tests de^ with equipment than^ras 
"generally used in a variety of ta^ and could be tested out of the specific equipment context. ThaT^the 
test subject does not have to icnow anything about the AN/APN-147 Doppler >^ tojo take these tests, 
^ich is not the case for Items in the spec^ test'equipment category. ^H^r 

A satisfactory correlation wasjound-fqr^e tests in this category. ^^^pPMf^^^^^^tch 
encountered, the subject was abfe to s^t up the controls of the TV-2 propedy for eacK problem (symbolic) 
- bu^^waTnot able to correctly^ interpret .results: (peffonnahce).^(It must be pointed out that this .device and ^ 

the^l890M: Transistor Tester were diffejrent mpdcls^f deWces'tiian the subjech^a4 been trained on; 
^ therefore, they ^had to orient themsely^ to these devices based on what th^ learned of the others.) II also 

shoidd bcL noted that this was.an ex^emdy limited "validation^ of g^nend^^tequipnient syn$r(^^ 
: ^^jFpre such tests can be accepted as ade^liatc^i each type of test must be^^v^^ full scale validati^.pn its 
'*"Ipwn4I1e^ts^This would require four. sqpapti^ validation efforts^ one each for the PSM^ Voitmeter^ the 
S^^S^f^B^^fekfa^nic dsciiioscbpe, the 1890NtJir5u^.istor Tester, and the Ty-2 Tube Twter, ' . 

* """^y SpecttQTcst &iuq>meiit (<l> - . Ji^ Th^bcc^al equionient tests dealt wiA e^ipment that ikd .tobe 
used ia tiieH:oittext!of a particular equipmen^i^iment ot4>rocedure. These results were less ssKisfactp 
than^nticipate^r ^,H^er, one of the nds^n^h^ may ha»fi beerr based upon the test subjects atWude. 
raster thdn the test mateiials. Ifis inatten&bn^^tall in. the JJPT caused the failure, yMe his Imowle^d^e 
of what to do allowed him to satisfactorily-co^lete the syniti^plic test. If his results had matched, the phi 
coefficient would have been .67 tostead of \ ^ , \ \S ^ • . \: ^ 



Aligii^mnt and AdjiStment Toito ^^j^^^tit^^is of the tests jn the alignni^nt and adjusMent . 
areas were ^nibined since the two surt as use uit t|s1^^i5triitegy;.4 rnodemtely stmng co ^ 
attained between the symbolic and performncd^^ ^as, sugg^^lingihat!bi^^ ^ 

are measure the same sets of sfdlls. In review^^*^Y^ur insUii^^s in wldch thelresults diffe^ 
.thejtwo test, forms, it was found that three testoui^optj^^as^^ J 
version. This ]rnay^have been partidly duetto tlu||:^^^stniie^%^^ 
''found that applying the scoring crftej^/^pr the^:^^e^;^eet dek^tr^ting^t^po^^^ 
(test equipment to radar, radar toft'^lwrne^^fi^^ llSxl 
been for Conventionally trained personnel." Tnis3^s^)|tt^^^^^ 



extraneous cOimectipn^. The subjects tended to mi 
uot^ets as weS. For purppses^of^^s ptpject, it ^as d 
oorrect cotmecttons wei^ made, rather than on the e 
required to determine how much ^^noise** of this natui 
incidental errors ^ch would not particulady degrade pe: 



_ id cpnnections/^ut^n^mwr of 
l^^the b^olt i^e^er 9^^^^ 
omiectiofls^l'urther W 
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rt<ubj«ct floun4^ring versus; ' . ikll^l 



Troubleshooting Tests (<t> "^-33). * Results of performance testing and symbolic testing in the area of 
troubleshooting shoi^ed a negative correlation. The specially trained novice .peisonnel used as test subjects 

* ^had difficulties. with both types of troubleshooting tests - the criterion referenced JTPT as well as their 

symb<*c cpunterpdrts. As indicated eariier, these test subjects had been trained to perform troubleshooting 
tasks with^^l?PJPA\which provided step-by-step troubleshooting directions. The test difficulties experienced 
by theserVu^bjetti were of, at leasts four listed types; namely: (1) deficiencies in the FPJP^rovided, (2) 
deficien^ies'in sequencing of the .troubleshooting criterion referenced JTgT.in relation to the sub-tests 
. in tHe^ JTPT battery; <3) maintenance difficulties with AJN/APN-'m? - AN/APN-35 sy^em, and (4) 
difficMties with the content and administration of the bank of test equipment pStorials provided by Ihe 
troubleshooting symbolic tests. The problems in any of these areas were of sufficient magnitude to prevent 
a higji poative empirical relationship between the criterion referenced-TFPT for troubleshooting and theit 
symbolic substitutes., ' • ' ' > ^ ' 

■ . ' , ' • , - * . . ^' * * 

^ The FPJPA had not been adequately validated prior to the Alttis effort. As a result,the st6prby-step* 
guidance provided by these aids was not perfect. TOs resulted in subjects having difficulty finding the 
(Equipment troubles without regard to whidi type of test they weiy taking. The tests were evaluating both 
-^Uie FPJPA and the subjects' ability to use them. In most cases, it was difficult to determine which had the , 
"^greater deficiencies. - ^ - . / • 

The recommended sequence for administering the various sub-tests of the criterion referenced JTPT * 
battery which is described in AFHRL-IR-74-57(H), had nof been developed at the time of . this Altus . 
validation. The test subjects were given }he$e troubleshooting criterion referenced JTPT before it had beeri^ 
determined that they could or could tiqi use their test equipments proficiently. Observations of the A^tus . 
subjects taking the troubleshooting JT^ indicated that many of them had difficulty setting up^^their test 
equipments and obtaining accurate test point information after setting them up. An inaccurate lest point 
reading would send the subject down a wrong branch in his FPJPA and as a result, Ke would' TOt^dentify 
the trouble. The FPJPA, used in the Altus effort, did not teO the subject how to use his test equfpmfent. His 
training was supposed to provide this "know how.? The training program, did provide instru^ion an3 some 
practice in the use of teSt equipments^ b.ut evidently it was not enough. This would support the hypothesis 
that technicians should be ^*over trained** in the use of test equipments, if they are to be expected to use 

• them*^ effectively in' job situations. TItis experience in administering these troubleshooting JTFT fttrther 
supports the position that troubleshootfng tests Viould not be administered to a subfecrUntil it is 
determined that he can use his test equipment adequately and can perforh other necessary sub-tasks, such . 
yfls checkout and alignment, in a proficient mamer. , \ 

Even if experienced subjects ha4 been available /or the AltUs validation'Fthe same test equipment " 
problem would probably.have occurred. A? discussed in AFHRL-TR-74-57(II), .other research efforts have 
indicated that many experienced technicians cannot useitheir test equipments well . 



The MAC field Aops ate authorized to perform these depot alignmtnts; and they, also, have the necessary special 
test equipment. The test developers were not axjfare of ^this cajlabtUty. " ; 



^ f' '-''If • ^ . ' . 

' . j rj . Ip^the'adji^tqgot'^and.alignme associated with the AN/APN-3S computer, it was found that 

J , 'out ^of/the $eveaWs, five of 'the,associated procedures in the FPJFA used devices that were not included in 

/ j^.^'. f tlie;^;nib<^ic m^ifi^d^ Those for which symbcHic tests had been developed were a track resolver drive tester 

I ; .^ Ind'^'^irquit t^dJsster. These other items atp generally considered Depot level equipment and are not 

» . . #j^^n^>iTiially found in fi^ shops.* The TO calls for the use of a VTVM and different proceducb, 

.\ C / ^{i^h were tfte bast^ for the development of the symbolic materials. Therefore,:no results were obtained in ' ^ 

^ ^ i^tpscfiWarw. Jj • , 

% . ""'^ ^"i-" ^'il ' - • >• ' 

' "-.V;;^.* : "^nce Ihel FPJPAs did -not identify adjustment and alignment pr^pedures as separate routine^ as is 
^ ^ , done ^l tKe TO, it was necessary to modify the testing strategy for alignments and adjustments. Normally, " 
theselests required the subject himself to locate th6 technical data required (to support performance. For 
per$onner using the FPJPA, howevei^it was necessary to provide the exact page and instruction reference 
necessary to accomplish the specific task. Since the same procedures were used in both the symbolic and 
performance.tests administratidn, this change did not affect test results. ' , " 



r 



Another aggravating problem for both the t€st subjects and for the idministrators concerning the^ 
criterion referenced JTPT was. the unrdiamtty'ofthe AN/APN-147hnd AN/ASN'35. Due to difiiculfles 
in obtaining some critical spare parts^t the ''time of Aim effort, these equipments were .not in prime 
^ operating condition: At times, an unwanted trouble would develop in the equipment under test while aiest 
subject was troubleshooting for the "planted" trouble Jn some cas^s,thrs invalidated the administration of 
a JTPT. ^ ■ ' . • " . ,^y^ ..^ . 

The symbolic substitute tests for troubleshooting were difficult to administer because ofthefr 66ok 
display of test equipment pictorials. This difficulty was ex^rienced in their original **shake down-* ?t*t ^ 
Norton AFB and continued during the Altus validation. f • • " 

Another difficulty^ concerning these symbolic tests for troubleshooting concerned the number of 
pictorials dyaHable. The criterion referenced JTPT make use of actual test equipments. When developing 
these tests)' the ^developer does not have to anticipate what use the test subject wfll make of his test 
equipments. If the test subject desires ^ome test point Information whether he really needs it or nqt, he can 
set up his test equipment and obtain the desired information (provided of course he can use* his, test^ 
equipment). But the d^elopers of s^mUdic substitute, tests produced for Uiis effort were required to* 
anticipate what test point informat^n the test subjects would request and to provide a test equiprae;yt*, 
pictorial display of the required information for each test point. The test dev^opers had anticipated, that 
the test subjects would be experienced technicians having received conventional training jncf^usifw,^ 
conventional TO. The test developers therefore, provided the test equipment pictorials yMchji^^^SeK 
would meet the needs of such personnel. The selection of these displays was based on a troublesfldJ^^^ 
straitegy which the test developers believed would probably b^ used by experiehci^d pers<yn&l*:v"* i 

The test developers had not anticipated the troubleshooting strategies used in the^FPiPA:^i^ the 
test subjects used in the^Alius validation asked for many of the test equipment display {called for^f)y the 
'FPJPAs, the displays were not ayqUabfe. When a display was not avaihble^Ufe iA^Would tell ^e subject 
what the test point information was for the problem. This action did notprovide-tl^ test subjecf with the 
opportunity to misinterpret the test equipment display thus decreases ttr^ realism of the sym|)qiic tesf.^ 
^^ ^his action made the symbolic tests more like the tab tests (Crowder et al., 1954). The result was that some 
^ subjects tended to solve symbolic problems when they were unable .to solvc^the sarne problem in JTPT, In 
fact, this test effort was trying to improve on this possible shortcoming of the tab tests. Before this pictorisd' 
Jype^of symbolic troublediooting test-^can be used successfully*with ETJPAs, the troiiblesnboting strategies 
of the FPJp^^jnust be studied and the missing displays required by the FPJPA must be added to the bank 
of test equipment pfctorials. 

tI^ ascertain if the test equipment displays in these originals troubleshooting , symbolic tests are, 
adequate fpr experienced technicians, would require a trytut making use t)f such personnel using standard 
TO. This has never been accomplished.* When arrangements were made for Tactical Air Command (TAC) 
subjects for such a tryout, the troi^leshooting qrmbolic tests were modified. Instead^ of^attempting "to . 
overcome the administrative and content difficulties of presenting a test equipment pictpfi^Tfot each ieil 
point required by the test subject, the ^mbolic tests for troubleshooting were modifie<l to eliminate this" 
; important p;ctorial feature. These modifications are described in Section IV. Whether or not this was the 
proper decisidn. Will tne disfcussed in Section V. ^ 

IV. DESCIUmON OF REVISED SYMBOLIC TROUBLESHOOTING :t£^ V; 1 

Bftcknound ^ * ^ -^l ^ - - 

What iwas actually required instead of or in addition to the limited validjitioncof th^ sr^mbolic tests at 
Altus AFB was a full scale validation using a sufficient number of experienced eleptronic technicians as test 
subjects (o evaluate the effectiveness of all the symboliiTsubstitute tests. At the timfe, the Air ForCeflumam- 
Resources* Laboratoti did not have such exploratory devdopment money an^ mAG could norfuriii% 
sufficient experienced subjects for the testing time required. When a small amount ofMnoney did<become* 
avaflable infFY73, it was not sufficient for the required largfe scale validaltfcn even if sufficient experienced - 
^subjects could be found. Jt therefore was necessary to limit the scope of the follow-on effort, gincc tiic 

' symbolic substitute tests for troubleshooting had presentecl the most problems at both ^(orton and Altus 
Air Forc6 Bases, it was decided to.con(;entrate the limited avaflable fesourciiVon that area. The objective 
was to reduce the TA requhements contained iij the original design of the troubleshooting fests while still 

c ^ ^ providing a^wide range of potential information about system parameters to* the test ^subject. * . 



I The initial sylpbolic troubleshodting^ests were based on the f9lloWing premise^:.- / « . 

1. Maximui;^ pictorial fiddity \rtth 'live^Mroubleshootto^^^ * . ^ . ' . 

2. Maximuni simulation of "live" troubleshooting procedure'/., 

3*. Integrated testing of component skills. ^ ^ J* 

While these premises were (and arc) hi^y defensible as guiding principles for embolic test 
con^ruction, their implementation presented several problems. For example, they?mpr^/w/se (pictorial 
fidelity) me^t that test point readouts are ^own as dikwings of meter needle positidn or sco^ waveformsf*^ 
This approach resulted in voluminous numbers of drawings of meter faces for all reasonable readout values.'^ 

The second premise (procedural simulation) involved thele^t-silbject "asking questions^(c.g., ^What 
reading do I get *af this point?'*) in the symbolic test just ais he would with Jhe ^live" equipment fAs the 
tests were developeS, this meant a one-to-one ratio between test subjects and TA, because the TA would 
Jiave to show the test subject the appropriate drawings in answer to his questipns. ' 

• The third premise (integrated testing) meant that all component skills In trpubleshooting (e.g., test 
equipment control settings, meter reading, strategy of attack, logical conditions from test readouts, use"6f 
documentation, etc.) were being ad4ressed simultaneously. Moreover, each troubleshooting problem 
incorporated analysis ^t the system level, at the chassis Ij^vel, at a stage, and within a stage. TTius, whereas 
success on a proldeni^was ea^ enough dto inte4>ret> failure to solve a problem could not be laid^directly^n a 
/w/fci^ skill component or ascri^^^ ^ ^ . 

Since alkparameters. were documented in the form of meter an4 other equipment readouts, this 
presented a formidable amount of paper. To organize this andewect a t^ subject to readily learn how to 
sort throu^ it proved infeasible. Therefore, i,*^most li|cely** subseC was^^^Velopcd. IWs, however, geneiajted. 
a need, for the TA to be able to scan a test subject's request rapidly, aiidito detennine whether he had;tfiiK 
g^ameter documented appropriately with a standard display, dr whether it was outside thejbpuikdsof the 
rpiolirem^ and should be answered with' a standard reply (e.g., •"in^tpiemiii;^^ bi a.l^tion, the TA^^cd to 
refiectrihe dynamic-effects of the subject!s^acti^ on * the eqiupnientrilf ;.,a kibji^t replaced a faulty 
component :2md then je^ new-readlnp'pr symptom information, ^^^^^^ would present different' 
displays thanjiadibeen showhpre^^ J1 ' . ' . ^' * : . \ 

'Hus.pro^efdure for syi)abpUcally> testingliroub^^oo tin^ ve/y readistk^ and provides a high degree of 
overlap-.with7tHe4nfprmati^ tiie equi{)jrnent and^documentation to the 

technician ih^^the'cpurjie pi^a^^^ for equipment, both- prime and test, is, 

eliminated^and}l&e tijme:ipr^sof>i;U^^ the equipmenfinterface tasks 

are elimlnated^^fe an expclrtrf A bgcrafeg.^ a one-to-one basis, or at least on a very 

low subjec.tiprTA, ratio, isintrpducseSi Thejef that would retain as many of the 

desirable fea|Wes«;^s poss the,c^mp|e5dty a^^ TA reqirfifcmeiits of the symbolic tests. 

To pro^de lall of the^icst pdfiit.an4sym equipment readings posed the prospect 

of volumes of displays to^wppprt;e^c^ trouble^Aqpting problem^ WMe the paper costs and devi^lopment. 
.costs were weB within'rcltfOn/vi^eti'^^ to the equi|!ment replaced thereby), a possible. iinacceptabjie 
cost Was that |nq)ose(l oalhfe tej^t^^^fect. He wbittdjhave to spend considerable time learning an^i^ing^e 
ri^ssaiy indeidng systems to iK^^iriaterials sas .^eU ^ ^that required to actually rMrieve a^ plece^.of 
information. Jf this inderiiig ^stqtiwe too comple?^he teaming and executing of th^ test mecharifcj 
could become ^lather diffl<?3t scp^tc skfll requifement^at coul<| possibly contribute to test faflu^r^J^;, « 

One sc^utiori to the prbblem ^as suggested eariier; that of using a rapid access pn?/ec/<?r; rather than a 
book for the pies^ntaiiqi^of^h^^ informatidn. Tlds approach would, pf course, call 

for -a simpl^ indexing sysfemi^fdrUapirig the infpmation bank, in addition, a sufficient number of;mpid 
access projectors would ha^l^p^ltie added to, the ..test administrator's kit. As^n alternative to numeram 
displays it was decided to provide, the rea4ings in a more compact formaf that would proiride better acce- 
to.mbre information. While this was^ giaiit step away from the goal of realistic infdrmatioh^ispky , jj wm 
. fdt to be vvorth;trying id order to^jiieve an adnl|jjjstratively.,viable troubleshooting test.* ^ 
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Approach v -. ^ 

The redesigiv^f tlie symbolic, troubleshooting tests was aimediat retainiiig'a 
with actual^uipm^troublestibotm^^ ^preach retained me features of pre^ 
' way as to* alTc^.^be wrt lAbjcct fo.^csta^^ pwnli:oubleshQpting strategy; 
. record his strate^i a^^j^^ than woi^d formally 0( 

approach was to jtei^tiftS admiiiistratipii to1arge"^oups of test subjects, with 
TA, yet provide tl^^^ta ^at a t^hriician would normally require while keeping 
• manageabfe'si/e. 

To accomplish tliis;2iid to provide a breakdown of the ability to troul^leshoot at th^various system 
levels, each of the origirral4|oubleshootiiig problems were djvided.into three levels: (l^i^^^tlon to a faulty ' ;« 

^ inii, (2) isdation to a fautty circuit , and (3) isolation to a &ulty pi^ce/pai^t^ The leyels oi^e^h problem are ^ 
coordinated in that the Cmty circuit, group, for ex^ple, Level i is withiiLth^faultyiimit of Level 1 and 

- likewise the faulty piece/part of Lever.3, is within the faulty circuit gtbup1)f Levd 2; However, each^level is 
designed to.^^ administered mdependently of, the companion levels. Hiis.provides Cdegree.of flexibility in ^ 

_ 'it^tingfauitaspIationat^artioMar ? ^ . " * 

^ ^ The iSaterials r^ulred for eaih problem part are provided to the test subject inciudin^ the applicable 
equipment TO. .The materials provided are used, to depict various equipment modes arid conditions. The * ' „ 
.^§t subject is reqmrfed to us^ this information, the TO, and his ovm knowledgr-.of the equipment apd 
-^^-le^ the fault, A detailed discussiph.of the design of each problem part is prpvidedin the / 

foDowingSs&g^hs; * . \' • ' < ^ ^ ^ . 

P0^-tsqbtion to aMqor Unit (Bktck Box) Level In this^part the test subject is requiJlwto use the , 
--^"trouble^^ting ijnjcedure for^unit isolation. ThePmaterials pro^dded depict multiple equipment $et*upc 
' 'fck each st;!p in Act procedure, the test subject is asked to select the correct cqulpmeilt^sefiip and the 
^ observe the corr^poi^ding visual indfcations. He must Oien ^4eci^j^hethct,fte indication is normal; or ; 
abnormal and proceed tp . the c orreq)ondmg stq? (see Appendix A)^-|hfiapproaSfv is structured in the sense- 
that the fault isolation strategy: is dictated by TO procedure. Hpwew^ihfi,p?pblem aUow the test subjeci-^^ 
tomakea'mktakeandtocontifiucunfil.fiejcorrectshimsdfbriw^^ " - 

- Part II - Isolation to a Circui(^foHp f Stage} LeveLJn this pairt the.\^iest subject is again requited to 
use the TO.,Howeyer, hieT establishes his own' troubleshooting strategy. Ijfe itjold what major unircontains 
the pxoU^n^pid is also ^ven a set,of data sheets whli?h provide^voftageiiid wavi^foim'ifata at key points^in, 
the equipment. Based upon the^^en sympt6im,lie selects points* tp measure. Th^;data sheets provide the 
yiAtage or Waveform at the sefect^ppint^TheJest sub^ ihust then dete^mini^ whetfier this data is normal 
^,or aMonnal'and §clect his next gcjiht accordingly (see^App^ndi^c B), Thus, ^e test subject us^s his TO as he 
would ig/a.ri actul txoubleshddti^ problem with the ineai^urei!>ent$Jbdx« data sheet_s.He 

. continue^iKfe process until he hjt^sroled as far as he c^an with-^d»t|grovided wjiidhjwll be from one tp, 
three stagbs. bi addition^idlistin'^^&e^suspcct stages," the test sjubjecf is also required to log each point he 
.measured on the data sheet, -^in^^tfe^prpblem is structured In such:* niamcr:as tp allo^ l^st subject 
to mMce some ittistakes Which he ini^t make troubleshooting oi^theai^^^ -^^ ' 

Part III - Isolation to Piece/Part LeveL-Thh part is identical in appros^^Ki^^Baitai. The inatcrials 
provided include all ^of the data provided in Part II plus a piece/part^dlg^^rfi^eer^^ ^^<«c/parts &i , 

the faulty circuit group (see Appenaix C> The test su)bjcctcpiitinues to n^kelftieasuremcnts ^^\h^,:did\in:; ^x-i 
Part ir to isolate, to a smaH circuit to which he cai mov^r check tadiyidu^^^ 

com|>'(^ent fMKeck and refers ta the piece/part data for its cpnditipn.;Inprder tp pr^eVent; jniprd : r % 



tte^^e/paj^ata Aeet, dl- of the piece/part valufes jare coyered^|hroug>r7ar^ 



* ^ sorfieWhat siniflar to tab^test. The test 4ubj<jct must u^ prdcr:tade|e^ 




Test Validatian Pmcednm 



«*vn^#iAW/«A/l M«KlA/«*»'^a«lit iUk'ti^-i^^MUp mUtS^^^- - ^fJ 



The Tactfcal Air Command (TAfe) agreed to provide experieiiced ttibjects^nd thcfeessaiy support 
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troubleshooting tests%nd their companion criterion referenced JTPT were administe'jicd to several groups of 
experienced technicians to ascertain the empirical relatiorfship bet>yeen the twa txt)es of tests. The results 
obtained from each symbolic substitute test was vjalidated against the resul^ pbtaTned from its companion 
JTPT. . , — . • * ' 

• fifteen subj^.ts^ere tested at two installations. Eight subjects were available at Langley AFB, and 
sevens were teijt^a.'^Xl^ittle Rock AFB. Considerable difficulty was encountered in finding the type of 
subject dedyrf^f^ ;this evaluation. It* was necessiry to have a pool of experienced technicians, since 
inexperienced pepsonnel cannot troubleshoot sufficiently well to provide -useful data as to the tests* 

^ rffectiveness. Further, it was necessary that the techihicians be experienced specifically on the AN/APN-147 
Doppler Radar. Due to thf general manpower shortage that was being experienced-by thcj Air Force at the 

' same time as the test tryout, many of the.personnel that were needed for testing were on extended TDY as 

a means of overcbming the effi^tsrj&fthe^manpower problems. A further restricting factor on ihe number of 

subjects was the length of the time required for testing. Actual troubleshooting problems require several 

hours for an individual to' solve and also require the use of an operational test bench in the shop. The 

symbolic test^.i^,guired almost the same amounf of time,rSo\hat for a man to take two problems via both 

the JTPT andHsymBqUc. tests r^ulre<I Vcomplete shift. Typically, men were made-available for testing at 

the rate of on6 per sl^vThelotal time required to test the 15 men was appro3cimateIy 100 hours. 
" ♦"•*••* • . ,^ ^ ' 

Each test subject was given" two troubleshooting problems to solve.. lii the JTPT test, a faulted 

component wSs placed into'the ra(Iar and the technician .^^ ins.tnjpted to Snjf problem 

consisted of two parts and the second one consisted of th^jjft"£^ • • ^ 

, / 1. Isolation of the chassis or bbck hox^yHielicontam^^^ 

2. Isolation of the stage or inodule in vMch the fault was located. 

3. Identiftotion of the piece/part that was faulty. . " ^ 

Probleri^ that 'Wer^vbaseB on faulty tubes wer^^ded as two-part problems; since a tube is 
consfdered to be part' of a stage qf an^dectronic circuit. TKe test subjects were instructed to work in each 
level until they could identify the location oflKe problem at that level. That is, in Part I they were told to 

. identify the chassis in which they suspwted the problem to be^fo^each subsequent P^^t of the problem 
they, were started at the correct ppint. Ihis was done to facilitate testing troubleshooting ability at each 

^level.. Previous efforts in which this was not Honejesulted m much lost time as test subjects pursued faulty 
a&umptions. . •. " > •♦-"^•^^ • ' ^ 1 ' « 

The symbolic tests were apjninistered in the same fashion. The test subject was instrSbted to identify 
" " the location of the fault at a particular levt^and s^op: He was then started on the next part at the proper 
, chassis ou stage* JTre ^rpblem presented to tkfiest, subjects in the ^symbolic tests were the same ones that 
. he received tJhe performance test. This was not ma^e known to the technicians, /however, and none 
showed any fec^]g^{iQQ,.that ^y were working on th&^e problems in both tests. ' 

la administering the tests at Langtey AFB-;;it«was possible to counterbalance the test administration ' 
so thathalf of the test subjects took the performaiT6e Version of the problem first, while the pther half took 
the symboUcr^Sion first. At Little Rock AFB, however, this was not possible since access ^o/the bench 
'-i^'quipment for the JTPT could not be progipimmed as readily as at tan^ey AFB. ^ 

' While'an"'^^-i46i4bjects wpre experienced in their primary AFSC, the range of experience on the 
^^^<ji5^PN-147 was qiiltB^^ide. Since the. doppler radar is only one :of sevlral major electronic systenjs 
c6vered under AFSC 328X4, an individual can be quite experienced ia'^sqme other system^ of the. AFSC 
-.'y^thout havir^ experience on the AN/APN-147. Table 11 shows the breakdown by test subject of the 
!^ijio|^ of service and tke months of experience on the.doppler system. Even this does not reflect the fact 
^^"^^Qme. people with adequate^length of experience may haye been. doing only flightline maintenance, 
which me^ms^ihat thej^onlj^ replace black ^Qxej in the aircraft and have hot been involved in the, shop level 
of maintenance: This'^njeiins^ttjey have tfi^ their total experience^in solving only the first level of the 
problems contained in th^^csts., - ^ 

^ The two .f<prnil-^f the. t^st were graded in the same maimer. For each level, the techniciari was given a 
j^U£^(+)Jife|^one^B)^^ thcA*$}ispect component, or a minus (-) if he did not. For the t6tal 
pToblehi lie W8S^^ a'luus'Snly. jf'he correctly completed each level of the,/ problem. While a record was 

- ■ ' ■ ••■ . • - 




kept of the parts he checked and the sequence in which he checked them, as^ell as the tcst^equipment that 
he used, this information was not used for scoring purposes. 

Due to the. time each subject; was avaflable for testing, the number of problems per test subject had' to 

be restricted to two. Since consideritte testing wj^ conducted within each shop and there wds no control 

possible over the nature and extent of contact among test subject^, it was necessary to give different 

problems to gu?r0 against compromise of the problem situations. Further, no feedback was given to the test 

subjects as to whether their solutions were correct until ill testing had been completed. Naturally ^ when a 

performance test was Correctly solved the technician had.a. strong idea that he had conectly identified the 

fault. There was also such inherent confirmation availabfe for Part 3 of the symbolic tests: This part made it 

important that the testing stirategy not be obvious to the test ^ijibjects. 
* *' 

Table 1 L Descriptioii of Test Subjects / ^ 
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Subject 
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Month* ' 
Service - 


Months of 
AN/APN-147 

Radar 
Expertenoe 
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SSgt 


78 ' 


36 
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sgt 


44. 


... 28 
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24 
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AlC 


19 V 


8 




5 . 


SSgt 


66 


66 


' V 
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AlC 


2i - 


-10 
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22" 






8 ' 


TSgt..- 


209 ' 


' 28 
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SSgt 


47 


36 




iO 


SSgt 


69 


21 
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AlC 


' i- 23 


8 




12 


Sgt- 


' •■ 43 ■ 


'24 




13 


AlC 


20 






14 


SSgt , 


85 






IS • 


•TSgt 


220 


None 





Since the objective was to compare test forms, the fact that* different subjects were taking different 
problems is considered to have minimal effect on thej:omparison results. But the fact-that Problem 4' was 
apparently more difficult than Problem 1 would be oT concern for comparing the abilities of the subjects. 
In that case, all subjects would have to have taken the same or demonstrably equivalent test problems. In 
this study f howe.ver, the symbolic test validity was b^iiig verified, so that each problem -pair is an 
independent set. The reasons f^^tfj^g different problems, as stated above^ were considered, therefore, to ' 
outwei^ the reasons for giving tnnMfe problems to all subjects. 7 ^ ' * 

Result ... 

To determine the extent to which the symbolic tests produced tl^e same results as the criterion JTPT, 
two types of statistical comparisons have been made between the nuniberof times, the two testing fomiats 
produced the same results. The first comparisons are made in temis of the tetachoric r statistic {r^) and the 
second, in tejms of the phi coefficient statistic (0). The 0 statistic was used in presenting the results of the 
Altus vajiditions reported in Section III of this document. In the Altus validation, all of the categories of 
synibiSlIc substitute tests incluciing checkout, spidering, remove/replace, use of test equipment ^ alignment 
and troubleshooting were considered. The'0 was used witii, the Altus results because the number of- 
comp^isons for each category of tests Was, in most cases, very small. 

The TAC validation, on the other hand, vww concern^ only with the troubleshooting category and - 
the number of comparisons between the symbolic and" criterion performance tests wa^ greater. The r^ 



statistic was chosen as the primary vehicle for presenting the results because it is a somewhat more 
sensititive statistic andjhe table used for its* calculation (Table 12) provides a jnuch better basis for 
discussing the results, lliis'table stiows passed and failed as'wdl as matched and unmatched results. The 0 
presentation, however, has been included so tRat the*results of this follow-on TAG validatfon can be 
compared with the Altus validation in terms of the same statistic* The comparison of the results of TAC 
validation in terms of the two statistics also is of some interest. 

Tetachoric r Analyses. Table 12 indicates the number of problems in' which the results \^iere:the same 
■ on both formf of the |ests and the number in which the results were different. The numbers in this table 
reflect the troubleshooting tests for all three levels of troubleshooting - chassis (black box), stage, and 
piece/part. 

Table 12. Gimparison of Overall 
Troubleshooting Test Results - TAC 
Validation (r^ = .68) 
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* Total 


14 
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These data represent the 60 tests taken. by th#^t5 subjects, 30 performance and 30 symbolics. 
Comparing the two tests t^en by the subjects on the same problem results in 30 pairs o^ scores. Iii 11 
cases, the technicians tested correctly solved the troubleshooting problem on both. In 11 other cases, the 
technicians failed to solve the problem on either test. For purposes of this study, the ability or inability to 
solve the problenw was not of concern. The goal was tqi^produce symbolic tests that would yield the same 
results, as performance tests. Thus, there were 22 pairs (a + d) out of 30, in which matched Vesults were 
achieved or in 73 percent of the ciises. A total of 16 performance tests (a + b) and 14 symbolic tests (a + c) 
were solved correctly. There were 3 cases in which technicians solved the symbolic version, of the tests but. 
not the performance, and 5 in which they solved th^ performance version but hot the symbolic. 

An r^ value of .68 was obtained with a standard error (a/^) of .288 so that the r^ value is 236 times 
th^^ndard error. For complete confidence in^the correlation, r^ should be 2.6 ti^esaf^. Ideally, also, N 
should be larger to satisfy the conditions for use of r^,1?owever, this was not possible due to the limited 
pool of Subjects. : 

' Given these restrictions, the results still indicate a positive relationship between the parameters of 
performance being measured by the criterion perfomiance tests and those being measured by corresponding 
symbolic tests. * ' \- 

Table'^13 shows the number of correct problem solutions by test type and test, problem. This table 
indicates that vMp problem dif^culty did in fact apparently vary (althoTigh the item difficulty versus 
abili^ interaction cannot ^e separated) the pattern of results was stable from problem^ to proUem. As 
discussed previously, the degree of agreement between results^of the two* tests is the primary concern rather 
than the relative difficulty of the individual problems. * ' , -^Z.... 
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Table J 3, ^i^mber otCpnect Solutions by Test lype^and Plrobleni 
Troubleshooting ProMems : 
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Performance 
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. 1 




•4 
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2 ' 


Symbolic 


3 


2 




0 


. 3 , 


= 2 


1 





Ylott. — n ~ number of subjects taking ea<^ problem* - 



As a means of^sessing tl^e influence of each pari or level of the symbo]ic test$ on the overall results, 
the results of technician performance on each part is presented. Forthe/irsf levd in the troubleshooting 
procedure, location of the faulty chassis or black box (i.e., identification of the Receiver Transmitter, 
Frequency Tracker Antenna or indicator as the location of the probteni), the contingency table of ijesults is 
given in table 14. ; ^ ^ 



' Table 14. Results of Chassis tor Black^x) Isolation 
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(d) . ' 




Total 


25 





(The data in Tables 14, IS, and lo^ie not cumulative but independent.^That is, a persoi^ failinglfatt^I ,^ 
still^ took Part IL Failing I and/or II» the technician stitt took Part JO; In computing tfie total test scores; 
previously pibented in Table 12, all requited parts had to be correct to g^t^a;Con:ect score oq the' total 
'probletn.y 



This yieldl an Ccjf -81 , The symbolic: test residts matche^d^^^e criteriotiperformahc^ t^t results in 26 
out of 30 tiin«i^or^m>^'87^^^ of thp cases. However; m can be »e^^ 

considerably more technic|ans successfully, performed- tMs- level • than pausseldf th^' complet^ tesit 
Twenty-seven oiu of jtfie 301imes, subjects were abie;to* pass,the criterion^JlPTt Much of the fli^t linip^^or 
organizational m^tenance Work w jconoemed'only with tMi level in' tro]Ubledj60tl^^^ /^S-^ 
finding a^d feidadM .tjie defective cnakis' or b6ck^^ 

expected,' sufccess to l^ating the faulty chassis is a .nci^essary bu(.hot suffibfe^^^^^ con||iHon fqr-prpl^c^^^^^ 

solution -in the Field^op (intermediate maintenanc^.VSnrUla^^^^ onc!s ^fli^fy toff^ 

containing the prol4em omnot be used as an indtk^i^^hmy^^ soivc.the problem. , ' : JT -Ht?^® 

The second level of the problem was 4o:lo^ 
The contirtgenqf table fo' these tesults js irTable l5,^^hf^^ test^reste^mat^ ' 

pcrfoirmance test results h]i20 <mt of 3pJmes or in d7*^^^^ ^ 
level of troiibleshootingii^^'out of 30 
; is, pks^the criterion JT^ 




Table 15, l^esuHs of Stage Isolation - TAG Validation 
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12 •= 
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The third piece/part fault isolation, was contained in only half (30) of the troubleshooting problems, 
v^ch gives an N of is matched pairs rather than 30 as in th^ previous test portions. Table 16 shows the 
results obtained. The r^ value is .16. The symbolic test results matched the criterion test results in 8 out^f 
15 times or in 53 percent of the cases. (In only 5 out of 15 attempts were the subjects able to isolate the 
defective piece/part; that is, pass the criteriori JTPT.) ^ 

^ ; Table 16. Results of Piece 
Pat Isolation - TAC Validation 
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Total =^ 
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7 15 



It is apparent^ then, that the third portion of the symbolic troul)leshootihg tests operated to reduce 
the oyeiall test correlation results. Iwwas also the otdy part of the t^st in which more test subjects solved 
the symbolic test than the criterion JTPT. Renneme^^flbrts focusing on this portion of the symbolic tests 
are indicated, therefore,«lihd could' possibly increase overall test correlation to a higher level. 

Compiinon of Results of Ahus and TAC Validations Analyses). As'stated eariieV, the Altus 
validation used novice subjects and was concerned with all the categoriesr of symbolic and criterion JTFT, 
whereas the TAC validation used experienced subjects and was limited to the trouble^ooting category of 
the JTPT and their symbolics substitutes. The symbolic troubleshooting tests had been modified betweehrL 
the Altus and TAC validations. Table 10 in Section III summarizes the resulted of the Altus validation iir 
terms of and ^ statistics. The formulas and procedures for obtaining these statistics also are presented m 
Section in and are not repeated here. The necessary contingency^ ^tables for determining the ^ and ^ 
statistics for tfl^TAC validatbn are presented in Tables TaUes 1 7 through 20. ^ 

faUe 21 repeats the information foufMcl in Table 10, Section in concerning the Ahus validation(^^Kl^ 
it also includes the and 0 residts for the TAC validation calculated from the data presented in Tables 17 
throu^ 20. It wS be noted that there is a decided ^provement in the condatlon obtained in TAC for ^e 
overall troubleshooting category ( .47) and the ^ corrdation previously obtained for the siame category at 
Altus (-i33). Tills improvement could be attributed to sevend variables,, such as the difference in type of 
sumects, a larger number of subjects and modification of the symbolic tests for Irouble^ootihg; The 
criterion JTPT» as wdl the ori^nal fonns andL modified forms of the ^mbolij£s> were developed for 
personnel trained to iise standard .tedmical ordeo. The TAC subjects were train^ and experienced in the ; 
use bf sudi tedinical orders, the Altus subjects were trained to perform troubleshooting using the , 
step-by -step procedures provided in^their FPJPA. 1[1ie symbolic tests did not contain pictorial displays forff 
all th Mest information called for by the FPJPA. Hie TAC subjects were experienced in the^mi^ntenance of 
the Ifcppler Radar System - th^..AN/APN.147 and thc,^/^SN-35/The Altus subjects vverendvices With 
the United training described ead^eK lliere were oviy^mnt pairs of tesb tried ^t Altiis whereas there were 
•30 in T^. Conadering thise vamWes, it is^iinpossible to ascertain jibw much of the improvement can be 



attributed to the ihodification of the symbdics. Btit it definitely can be concluded" that symboUc 
troubleshooting tests show promise and their development should be continued. . ' 



Table 17, Overall Tioubleshooting 
^ ' («=.47) 
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Table 18. Tioubleshooting Chassis 
Level (Bhck Box) (0=. 73) 



RtsuHt 

MatehMt 


Reyuitt 
Differed 


Results 
Matehed 


Results 
Differed 


Expected Frequency ' 15 
Observed Frequency 22 
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15 
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Expected Frequency 1 5 
Observed Frequency 26 


15 
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Table 19. Troubleshooting Stage 

Level (^ = .33)' • ' 


Tgble 2Q. Tioubleshooting Piece 
PSart Level (0 = . 07) 
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Expected Frequency 15 
Observed Frequency 20 
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Observed Frequency 8 
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Table 21. Indi^ 


igXdmbined Results of Altus and TAC Validations 




• Test Area ^ 


N Results 
rtiirs Mitched, 


Results n \ 
Differ X 4> 





Novice Subjects (Ahus) 

^Checkout . * 4 

Remove & Keplace. 14 

-Soldering Tests "^"^ ^4 

General Test Equip ' ' 6 

Special Test Equip . ,6 

^ioiipent/Adjustment 1^ 

txmibleshooting ' ' 9 

Experienced Subjects (T/kG) 

Ov^iaB TroubleAootinp^ 30 - 

Chassis (Blacic box) 

Isolation 30 

Stage Isolation 30 

Piece/Fart Isolalio n 1 S 



4 
10 
3 
5 
4 
15 
3 

.22 

26 
.20 
8 



0 
4 

2 
1. 
2 
4 
6 

8 

'4 
10 
7 



4.00 
2. 57 
0 

2.67 
• j67 
637 
I'jQO 

6.53 

1633 

. 333 
.07 



1.00 
.43 
0 
.67 
33 
'58 
.-33a 



.47 

.73 
33 
.07 



.68 

.an 

.46 
.16 



^This negative corrdation was mob ably due to a number of deficiencies sucKas (1) deficiencies in the ^xiUy P^<^* 
duralized Job Performance Aids provided tb^ subjects^ (2) deficiencies in the sequencing of the troubleshooting JTPTiiv 
relation to the sub-tests m (he JTPT battery, (3) maintenance difficulties with the AN/APH-147 - AN/ASN-^S system, and 
(4) di^ulti^with the content and administration of test equipment pictorials provided in theori^nal troubleshooting 
symbolic tests. r . * ^ ^ . . 



Hie r^ ^correlations for the TAC validation have aisp been included in Table 21. They indicate the . 
same pattern of relationship between the syrKbcdic and gerfotmance tests. However, the r^ figures are larger / 
than the ^ figur^pThis should be expected since the is a more.^nsit{ve statistic. Both statistics indicate 
that symbolic 1 csts: for the piece/partievel are in nged of further refinement . V' 

^mbo6c Test Clarity. There was a grea^ difference in th^^Qity <^t^^ technicians to gmsp' ibaff 
testing concepts used in ^e symbolic troubleshootuig tests. The m^^fit)^(^owed the instnicUohs ^dihm ^ 
di^iculty once the nature of the test im reviewed with them. AidcUtion^'i^^ was pro^ded^'m 

several cases so that instructional misunderstandtag/is not consideredi to haw|«5en a factor infliwndng test 
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results. Several, however, were not able to conceptualize the fact that all of the information they' needed 
was present in the materials just as it would be in a faulty radar. They could not construct a search question 
or select a reasonable starting point for isolating the problem. In other words» these subjects did not 
demonstrate that they had an effective troubleshooting strate^ at their command. AO th^y could resort to 
was alinear search of the system parameters given. ' ^ 

This analytical approach is not' always used in actual field performance where the system is avaflable ^^ 
for immediate feedback. In some cases these subjects could solve the performw|ge version of the problem . 
based upon experience history or by replacing components untH the faul|pjbv located. The history 
approach is sometimes very effective. However, as tne realiability of replaceable components is improved in 
any system, the history approach becomes kss and less effective. Repkcing components untU a fault is 
located is an extremely undesirable form ofSroubleshooting, It loads the maintenance system with good 
. components that must be checked out. It generates unnecessary equipment handling, paperwork, and 
benchwork. And experieiKe has indicated that many times these good components are discarded as faulty. ' 
As a Vesult, an extremely good case can be made for every maintenance technician being proflcient in the 
use of an effective analytical troubleshooting strategy^, such a strategy is required by the symbolfc tests. 

VL EVALUATm.COMSlENJS^ ,1 

' I 

Considerations for Future Develofiment and Implementation - 

The efforts reported in AFHRL-TR*74*57(II)» as well as in this volume for the development and- 
tiyout of criterion referenced JTPT and'accompanying symbolic suj^titutes have been plagued by the 
limited availabflity of experienced subjects. The criterion referenced^JIPT described in Volume II are 
worthy of admuiistration in their present form, but further polish ^uld probably improve them. This can 
. only be accomplished by many more administrations of these J1PT under the guidance of experienced test 
; developers. In spite of the limited number and inexperience of the Altus subfects, the'symbolic substitutes 
already developed for checkout, remove/replace, and use of general and specid test equipment have shown 
a hi^ degree of promise^ but require more extensive vdidations and refinements, pie aligmnent and, 
trouSkshooting symbolic tests have shown promise but both categories require a great, deal more 
refinenSint before they should be validated again. No further work is recommended on symbottc substitutes 
fof soldering. The exphmtory work has identified major and minor problems of concern for fUture work It 
i^abo provided a Imis for planning of future work ^ 

^ The success of any future developments or refinements of either criterion referenced JTPT, or their 
symboUcT^ubstitutes, will depend on the availability of sufiicient funds, expert test development perspnnel, 
andxjualifled subjects. All of these have been in short supply. Although dl of the necessary development 
and refinement work is'considered imjpj[i£tanMt"waddi^ inAossible to accomplish all of this. work at the 
same time consideYing the available^'resources. The recommendations made in Volume 11, for an orderly 
development aijd implement jtiofif program for JTPT, must also be considered for any future development, 
of symbolic substitutev^^'^piiically ^alid symbdic substitute tests cannot be produced for any job activity 
untfl good, adjpinistfable criterion referenced JTPfare available. 

f^^^ltfUe- Volume II recamMendations suggest that JTFT refinement and implemefitation should stq(:t with 
general t^st equipment. Since all electronic technicians use general Jtest equipment^ tlie proper 
implementation of this action should result in the greatest gain to the Air Force for the least^ampunt olf 
effort and money. So logically , any future work in symbolic tests should start with thi^amc area. As JTPT 
• are reflned and become available for other maintenance activities, appropriate work on companion 
symbolic tests can be pursued. The comments that follow are made for the purpose of helping concerned 
andinterested people or agencies-to structur*e future work on symbolic substitute tests concerning: (1) the 
use of general test equipment, (2) other strait tline tasks (checkout, rein<^e/replace and u:e of special test 
equipment), (3) troubleshootii^gi and (4) alignment. ' - . . . ; t " 

Symbolic Tests^^r General test Equipment. In addition to the op^fmiional considemtiqris for 
choosing the use of g^ral test equipment for iniUd fottow-on work cqmerning symboUc: tests, such 
equipment is a very fertile area for symbolic presentation for prcdriing as well as testing This is especially ^ 
true of the .yoltohmmeter and the oscilloscope, Most of Ihe dffficult a^tiv^^^ concerning their use arc 
perfonned;on their front panels. And panels can M very easily yepre^Kted t^y-^ic^^^^ 
require, the manipulation of switches and knobs; behaviors ^hich ar^' within the nonn*^epertoire of most 

• , * ; - • ' ' \ ' 
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Americans. The difficult part is learning the proper sequence in which these manipulations should be 
jperformed. They all require the ability to read displays. All of these behaviors, with the exception of the 
actual manipulations, can be simulated by pictares. In addition to the development of symbolic substitute^, 
for JTPT, extensive use of symbolic representatives should be considered for practice exercises for training 
purposes^ Their use may greatly reduce the requirement for actual equipment for both personnel testing 
and training purposes. . ' , ^ 

The Aim validation of symbolic tests for general test equipment was extremejy limited Only six test 
equipment symbolic tests were administered and the results were lumped for analytical purposes. After 
another refinement, the symbolic tests for each test equipment should be validated in its own right using a 
sufficient number of subjects for each validation. Such validations are necessary to ascertain the degree of . 
empirical validity for each test. The validation exercise, if properly structured, will probpbly provide a great 
many insights for the use of symbolic techniques in the training process. ^ 

Symbolic Tests for Other Bask Straightluie Ta*s. Whatever is learned from the propped work on . 
general test equipment can probably be applied directly to symbolic tests for special test equipment. Much 
of Mdiat is learned can probably be applied to checkout and remove and replace activities. For- safety 
reasons, checkout is an extremely Jmportaht activity conceming any system. We should have means to 
determine how effectively these checkout tasks are performed. Both criterion referencQj} JTPT and 
companion symbolic tests should be developed for several key equipments for an AFSC such as 328X4, and 
should be validated on a large scale. ^ " ^ o / 

SymboBc Tests for Troubleshooting. The original attempt to validate symbolic tests for 
troubleshooting at Altus was unsuccessful. There were several reasons for this which have been discussed 
eariter in Section III. As explained eiriler for the TAC validation, the symbolic troubleshooting tests were 
mc^ed to remove the pictorial presentations of test equipment information, Volt and dm information 
w^fumistied in printed form in tables and waveforms were given. But the subject was not rc^quired t6 ' 
obtai^any of this information by interpreting a picture of the test equipment front panel display. This- 
modification jnade these troubleshooting symbolic tests very similar to the tab tests (Crowder et al., 1954). 

The tXc validation indicated that these modified symbolic tests for troubleshooting did have a high 
degree of empirical validity for measuring a subject*s ability to identify faults at the chassij^or black box 
levd. But the symbolic tests did not do as weU for fault isolatton to the stage level (within a black box). 
And at the piece/part level of fault identification with^the modified tests, the validation indicated that the 
embolic tests had a very low emgirical relationship with the criterion JTPT. The piece/part level is the, 
most difficult and expensive level of hiaintenance in terms of spare parts consumption. These weaknesses 
must be corrected before symbolic tests for troubleshooting can be considered successfyl. 

In the opinion of the writers any future refinement work conceming troubleshooting symbolic sub- 
stitutes should reflect the following considerations: ^ . ^ . ' 

1 . The original rationale for the development of symbolic tests was formed after a study of the 
resulb of the tryout of tab tests for measuring ab|ity to troubleshooting (Crowder et al., 1954). The study 
~of the tab test data indicated two possible weaknesses. Fhst, the Crowder tryout assumed that a subject 
taking the criterion performance tests could set up his test equipment correctly and olftain correct test . 
equipment readings* The readings were given to ths subject upon request. fStei a. review of the Crowder 
study, the writers were of {He considered opinion (based on Crowder's and other research) that the subjects 
probably could not use their test equipment too wdl. As a result, even^oug^i a subject cbM find the fault 
correctly on the tab test, he was not a{>le to find the same fault In his actual equipment because he cou|d 
not use his test equipment. ' • 

Second, approved or desired troubleshooting is usua^y based on a strategy or cognitive process. The 
, tab te^tt (as weU ,as most other paper and pencil tests of its Jdnd) also requires the use of such a strategy to 
successfully identify the fault. But in the tab test, the subject was not faced with the problem pf ^ 
interrupting his strategy to obtain test point information. He received it instantly in printed form and 
proceeded with his strategy, whereas, the requirement 4qftet up and use test equipment to obtain test point 
informatfon during the ^^riterion performan^ test' lactual equipment) provided many distracting 
interruptions or "clutter** in his strategy. ' 

. 2. A proposed solution to the^problem that a subject nlight not be able to use. his test equipihent 
was presented in Volume II. Each subject can be ^ven a performance test on the use of his test equipment • 



before he receives his troubleshooting tests. And, if the subject cannot tpa^liis test equipment test^he 
would not be permitted to take thetroubleshodting test. Due to the lack of money, time pressure's, andIKe 
limited number of available subjects, this was not done during the TAC validation. 

One hypothesis as to the cause. of some of the different results lor symbolic tests for the chassis, 
stage* and piece/part levels of troubleshooting is that some subjects were noit able to use their test^ 
equipments pbpeiiy. The chassis .level of troubleshooting requires littlruse of test equipn^ent. The 
0 criterion relej^enced JTPT and th^ir symbolic substitutes thus call for very simildr behaviors. The stage level 
of trbublesho'!!^^ requires some very common juses of test equipment. Ijf-this case, the criterion JTPT and 
their symbolic suf^iitutes are reasonably clqse. ^ The ^su bject is re^quired to make limited use of test^~ 
equipment in the criterion test, but^no us^ of test ^equipment in the^symbolic test. The piece/part level of 
trouble^ooting requires a great use of test equipment. In this case;; the criterion JTPT and the modifled 
syinbolic |^ts are far ajparrih respect to the test equipment bdiaviors required. The subject is required to 
make gre^^^^ of test equipment in the criterion JTPT-but/rto use^of test equipment in the symbolic test. 
This difference, of course* makes the symbolic test easier than^iljfe criterion. It should be noted that 8 out 
of the 15 subjects passed the symbolic test and only 5 passed the criterion (see Table 16). 

The above 'hypothesis should be further explored ini future wor* on symbolic tests. A sufficient 
, number of subjects should be giVen'the test equipment JTI^, and those*who pass the test equipment tests, 
^ould be assigaed to one categoor; those who do not, to anbther category. Both categories should be given 
the criterion JTPT and tljie symbolic tests for troubledipoting. If the hypothesis is true, the correlations for 
the stage and piece/part leyels should be improved for the -category of subjects who are able to use th^ir test 
equipment correctly. Th^ condations for the fcategQiy oP^bjects who cannot use their test equipments'^ 
codd be evealower^hant^ose obtained in .the TAC"^«^ . • 

3. The above hypothesis may account for pari of the variance,obtained between the criterion JTPT 
. and the sj^mbolic tests dicing the TAC validation. B6t it may not account for all of it-^if the distraction or 
**dutter'* tactSr*is of un^rtance.-As discussed earlier, the tjroubleAooting symbolic te^ts developed earlier 
wliich were used in ti^ Altus validation contained displays of test equipment, froiitr^ panels. The book 
presentation of this information to the test subject by thfc test administrator proy^^difficult. As stated 
earifer, these were eliminated wheit the symbolic t^ts^were modified for the TAC vafidation. As a result, no 
.data concerning the effect of this t^pe of "clutter* Yor tj:oubleshoonng has been obtained. The modified, 
.symbolic tests did require the subject to search his-technical^order ^^^uch is one type of realistic clutter. 

In 'Section 11 of , fids dotiintent,- me suggestion wa^ made that a random access i^ojec.tor^might* 
possibly be used in pla^of the book of displays. The authors are of 'the opinion that this suggestion should* 
still be tried. The dis^^s'^could easily be placed in a random access projector but the problem is the 
requirement For a simple Indexing system. Bi^ the test point information formats developed for the 
modified symbolic tests may provide the answer. Hie test point information (such aslSOvor 80J2) found in 
the cells of these TonrWould be replaced by the c^ oit numbers to be punched on the control panel of 
the-tandom access proj^or. • ^ • 

Tlte use 'of such arandom access projector progrmn isistrongly suggested for any future exploratory 
work on symbolic substitutes for troubleshooting, The^^evelopment should, also, include fUrther work on 
fust what displays are required There is reason to belike that there were not enough in tfie original effort, 
ftis certainly better to h0^e too numy available than too. few. / • 

It is Hypothesized ihat embolic tests tlfat include this type of clutter will result in high correlations 
between the criterion referenced JTPT and the symbolic tests for the piece/part level of troubleshooting It 
is suggested that subjects used in the validation of jhe test also be divided into tvk) categories; namely, 
those, who have previoudy passed, th^ perfornt&hce tests on the use of test equipment and those who have 

* Symbolic Tests for Alignment Tasks. :Althcugh alignment ,' adjustment and calibrating tasks are as 
important as troubleshooting, , it requires 4ess Xin^t Xo measure such activity .with JTPT using actual 
equipment than it ddSs to measure troubleshooting activities. In addition, what Is Jeamed about 
symbolizing "dutter" ^r troubleshootin^niay^help kvthe devdopment of synibolic tlfsts iFor alignment, 
activities. It-is the^eforji. suggested that further work or]i*syml)Olic tests for alignmenttasks, be delayed untfl 
after the troubleshooting-symboUc test problems are s<^lved. v * 



I^aintenance Qii^ity Consideration 

Althou^ the prtmary purpose of the TAC efforts were to validate the effectiveness of symbolic 
' substitutes, a secondary finding oi> quality, of trouMeshooting is worthy ofcdmment.lt is very difficult to 
find hard data concerning the true quality of maintenance jn the Air Force. Some studies concerning ability' 
of technicians to. use test equipment were reviewed in Volume 11. But all available data do indicate that the . 
'ability of technicians to perform troubleshooting could be greatly improved. The results of the TAC 
* validation certainly silpport 'this contention. Table 16 indicates that the experienced technicians used as \ 
subjects could isolate def(?ctive piece/parts only 5 out of 15 times, or only 33 percent of the time. Th^sam^. 
subjects were able to identify faulty stages only 20 out of 30 times, or 67 percent of the time (Xable 15).r 
'They could identify^black boxes in 27 out of 30 times, or 90 percent of the time (Table 14). ' \ . / 

This performaiice for blatk box fault isolation is fairly good. However, black box isolatioivfe the 
easiest level of fault identification. And,\X is very deceptive because if there is a large enough §i^)ply of 
blaqk boxes, the planes still fly, and it appears to the pilots that maintenance is fairly good. TMs supports 
' the contention made in Volume II that activities having the greatest visibility get the most attention. i4wi 
shop maintenance does not have near the same visibility as fli^tline. » ' 

But the "sleeper" in this situation is that once the black boxes are identified, 'they are sent tp the 
field shop for repair. And the ability of the technicians to find the trouble in the black bo)^es would appear 
to be very weak, resulting in very expensive repair for the Air Force. Findings sucn as these further support 
the contention that good criterion referenced JTPT should be availablejor the , purpose of gatnering 
. extensive h^rd ^ta concerning the ability, of technicians to perform imiintepancyactivities. the known 
presence o6 such JTPT in the field and the possibility of technicians being t^d, would probably motivate 
many technicians to learn to perform their key maintenance activities more efficiently. . 

* * • . • . * / ^' * • ^ 

VII* SUMMARY AND C0NC|;CS10NjS / 

In thfe effort, graphic symbolic substitutes were ^^veloped for each type of electronic maintenance- 
activity for which criterion referenced JTPT had jdready been developed. The JTPT are^d^scrib^d'in ' 
AFHIUWrR-74-57(II) Part I. The job activities include: ta) checkout, (b) remove/replace, (c)' soldering, (d) 
use of general and special test equipment, (e) aligr^adSust/talibrate, and (f) .troubleshooting. " y . 

I The hardware utilized as a vehiclCi for t^jfe development of all the^ tests^as'-tjie' Doppler Rkdar 
,AN/APN447 and its Computer AN/ASN-3 5/; ' , ' 

In \he development of ^ch of tlie symbolic substituies^^an atJemplt.WiM made to require the test 

subject to demonstrate as hearly as possible, the same behaviors a§ he would "betequired to perform while , 

takftg its companion JTPT. ' x " 

' x ' y . - 

The first drafts of these symbolic tests were given a^limited administrative try out in a maintenance 

squadron of the Military Airiift Command ^AC) at Norton'AF^XA. The subjects were technicians who 

had maintenance experience onlhe AN/APN-147'and the AN/ASN-35."Thi8 tryout indicated that all the 

tests as developed were administratively feasible with the exception of jthe troubleshooting tests. 

After this administrative tryout, all of the symbolic tests were refined based on the experience gained 
during the tryout. A full scale controlled validation was planned to ascertain the empiricd validity of each 
type of symbolic substitute. Due to schedule changes and unavailability of experienced subjects, such a 
validation exercise was not possible in MAC. . ^ - ^ - , / 

A more limited validation was substituted making use of- avidlable .novices with limited training at 
Altus AFB»,OK. These subjects had been given a fou^;.Week training program^onhow to use test equipment, 
how jto use. hand tools andi how to perform the mamtenance tasks on the AN/APNtl47 and AN/ASN-35 
usingVpec^ step-by-step maintenance instructions called Fully Prpceduralizeid Job Performance .Aids. 

J \ . During this extremely limited validation, bqth the criterion JTPT and the^ Companion symbolic 
' substltutes^ere administered to the^same novice subjects. The symbdic tests for checkout^ remove/replace, 
general, and special test equipment, and align/ adjust/csdibrate showed promise. The troubleshooting test^ as 
structured were still npt administratively feasible. The sddering symbolic tests, although administratively 
" feasible, showeji no empirical relationship' with the criterion soldering test. 
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The future development /refinement and implementation of symbolic tests are dq)end^nt on the 
availability of criterion referenced JTPJ^.- Empirically valid symbolic substitutes cannot be developed and 



The chief administrative difficulty experienced with the symbolic troubleshooting tests used, at 
Norton and Altus was'caused by the technique used to realistically simulate the use of test equipment. An 
attempt made to present all necesiary test point information as it would appear on the actual test * 
, equ^ment. This required one test equipment pictorial for each test point called for by a subject taking the 
embolic test. A book of pictorials developed. When the test subject in^ated a requireq[ient for test j 
point informatk)n» the TA would show him the appropriate picture in the took if it was available. The TA 
functbned both as an index^to the data bank and as a searching defvice. This proved a difficult if not 
impossible task. • , , » 

The use of a random access projector was suggested in lieu of the book, but tliere was insufficient ^ . <^ 
time and money remaining in the project. In addition, the effective use of a random access projector would 
require the development of a sim^e but effective indexing system. 

Wth the limited available funds, an attempt was made to modify the troubleshooting symbolic tests 
by compressing test point informatwn. Several forms were devdoped for presenting printed voltage and 
resistance values and simple wave form displays. This was a retreat from the realism of the original symbolic . 
tests, making them some^^t similar$to the tab tests; 

Anangements were made to validate these modified troubleshooting symbolic tests in the tactical Air 
Command^ing experienced technicians. ^fteen subjects were obtained by administrating the symbolic 
tests and their criterion JTPT at I^gjey AF6, VA, and U^^ 

The results of this? validation indicated an improvement in the adndnistfabHity of the tests. The 
empirical relationdiip between the symbdic t^ts and the criterion IT^ at the "black box" or chassis level 
of troubleshooting was exc^ent; at'sta^e levd.ithe relationship was fair; but at the piece/part level the ; 
relationship was extremely low. This validation indicated that symbolic troubleshooting tests have promise^ 

but more exploratory development is required. - 

' - ' . ■ ' . " ^ , \ 

-\0ne hypothesfe. offered:for these results is that at the black box level of fault isolation little use is 
made of- test equipment; as a result the'-realism of the symbolic t^ts is high. At the stage level of fault 
isolation, a Jirnited use of test equipmenttls. made, and the realism of the symbolic test is fair. At the v 
piccc/paft level, extensive use of test eqmpihent is made. As a result, the printed presentation of test \ 
..e(pji»nent information results in a large deviation from the job realism. A return to. the original pictorial 
presetitatibnof test equipment infonnat ion, the^fore, is ^ . , 

^ N As stated eadier, one suggestion for presenting test point information in pictorial form was the use of ^ 
" ^a tjtodom access projector, provided a simple indexing system could be d^eloped. The forms developed for 

\<cornpacting test point information for the trouble^ooting symboli^ tf^ts.u$ed for, the TAC validation 
provide a possible solution tp. the indexing problem. The substitution of jiccess code numbers for the ^ 
resistance and voltage values now found in the cells of these forms may provide the simple indexing system 
for obtaining the desired test equipment pictorials tapidly, 

\ A second hypothesis is that the modified trolibleshooting symbolic tests used in TAC may^e viable 
tests, provided the test subject is required to demonstrate his ability to use his test equipment before he is v 
permitted to take these troubleshooting tests. This procedure was not used in the TAC validation of the 
modified troubleshooting symbolic tests. 

Althdii^ promisir«, the symbolic tests for align/adjust/calibrate activitle*^ require more refinement. 
One special jui^a that has not been exploreaidequatdy is the tuning skill required in some alignmerit. tasks. 

The results from the soldering symbolic tests would indicate that no further work should be done on 
these tests. An JTPT on soldering is not too difficult^o administer - so, very little would be gained even if 
an en5)iricaUy} vaUd symbolk test could be developed. # ' ^ 

A secondary finding of the TAC validation was ^Tthe experiwldd technicians used af^^ts 
'could not troubleshoot very well'to the piece/part leveL Only 33 percerit of the defective piece parts were 
identified. TTiis findirlg supports other available hard' data in the contention that, in general, electronic 
te<^icians do not trouBleAoot very efficiently. 

' . . • ^ ' 

^ » .VOL RECOMMENDATIONS 



validated without JTPT.^The validation process also is dependent on the availability of a sufficient number 
of trained and experienced subjects. Unless sufficient money is provided, for test development and the^ 
availability of subfect^^ehsured, the development of symbolic substitute tests should not be attempted and 
is not recommended. 

V 

AD of the promising embolic substitute tests for straightline tasks should be given a full scale 
validation. But due to probable fund and subject limitations, the following priority is recommended: (1) 
general test/ equipment » (2) checkout » (3) remove/replace, (4) special test equipnleot, and (5) 
sdign/adjust/calibi^te; _ . ' 

Tl^e modifled troubleshooting symbdic tests used in the TAC validation should be revalidatedo^iag a 
larger number of subjects, who have demonstrated their proficiency in the use of test equipment by 
,^ccessfully passing the .test equipment JTPT. 

A randoni access projector presentatioli of pictorial test point information should be developed. An 
indexing system is suggested, which makes lise of the information forms developed for the modified tests 
used in the TAC validation. Such an index would be obtained by substituting an access number (for the 
random access device) in each cell of these forms for the printed test point information,now found in each 
cell. When the test subject punches the access number^ he will obtain a pictorial display of the appropriate 
test equipment which he must interpret. 

After such troubleshooting symbolic tests are developed, they should be validated and their empirical 
validity/:ompar44 to the results obtained in the previous paragraphs. 
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-APPENDIX- A 

PART I. SYMBOLIC TROUBLESHOOTING TEST MATERIALS 
- Isolation to a^Majpy Unit or Black Box'Eevel 
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. • ' ' . PART IB 

t 

TEST SUBJECT TEST INSTRUCTIONS 



Part IB: Isolation to Unit. ^ 

- 4 • ^ 



INSTRUCTIONS; 



a- Ensure that you have the Radar Set AN/UPM-147(V) TM and the ' 
f Allowing Information sheets: ^ 

t(l') Part IB Test Performance Sheet 

(2) Part IB Answer Sheets. • . ^ 

..b. Eriter your assigned identification number on the 'Test Performance 
Sheet. , ' 

r^llow the instructions given on the Test Performance Sheet. You 
ma^sSse any and all parts of, the TM tp answer the questions. 

• d. Whe^JI^ have completed all questions turn in Test P^formahce 
to the Test Administrator. ' 



c. 



/ 



PAST IB 



TEST PERFORMANCE SHEET 

» 



Technician ID No. 



'J- 



A. Read step 1 of the troubleshooting tableX Examine step 1 answer sheets 
and record the number of the answer sheet which corresponds to indicated ^ . 
switch settings. Note associated. digital counter indication and proceed 

/ to step B ^t' indication is normal and to step F if indication is abnormal. 
1. Answer Sheet . ^ . _ . ^ ' . 

B. Read step 2 of the troubleshooting table. Examine step '2 answer sheets 
and record the number of the, answer sheet which corresponds to Indicated ^ ^ 
switch settings. Note associated digital counter indication and proceed 
to step C if Indication is normal and to step F if • indlcatidn is/^bnorm^l. 

^ . 1 . Answer Sheet . 



f C. Read step 3 of the troubleshooting table. Examine step 3 answer sheets* 

and record the. number of the answer sheet which corresponds to Indicated 
J V switch settings* Note positign;^ associated Drift-Angle pointer and 



proceed to step D if indication is normal and to step H if indication is 
, ' abnormal. ^/ k^^' -^-t . . > 

1. Answer Sheet "i-i • . * * - \ ' ; 

> - • . • • . • - ^ ; " • "-^ 

t). Read st6p .4 of: the^ troubleshooting, talile. Examine step 4 answer sheets ; ;J| 

and 'record: :the .number of the, answer ^heet which corresponds to indicated t 

. - . . switch settings. Note position of associated Drift- Angle pointer and 

. ' . ^ . ^. . 

proceed to step 5 if Indication is' normal and to Btep H If Indication ;:^^^^^, J^^ 

\. abnormal. \ ^ / ' ' \ / ,>V:^| 

• i , Arisvrer*Sh'eet#:- — ^ • ■ «, ^ ■ : : ■ - ■ - . , •. ;• ,;;.:-::..§jS. 
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APPENDIX B 

"PART II. SYMBOLIC* TROUBLESHOOTING TEST MATERIALS 
IsolatioiiP.to Stage Level 
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PART H . 

* ■■.( 

.TEST SUBJECT TEST INSTRUCTIONS- 

Problem No. 1 • > * • 

INSTRUCTIONS* 

^a. Ensure that you have a TMand the following Inforaatlon sheets. 
(1) Equipment Visual Indications ' - ' 

• (2) EqulpmAt Test Point *DatA Sheets' 

.* * ■ . 

(.3) Equipment Tube Pin Data , ^ ' 

(4) Part II Technician Performance- Sheet 

b. Enter your assigned Identification and problem numbei^ on the Test 
Performance Sheet. 



c. A problem has been Isolated to the Indicator unit. Your job is 
to Isolate this problem to a stage or smill group of stages. 

d. You wlll.be simulating measurements on the equipments as follows: 



(1) Use visual Indications and TO to detl^^ where you want to make 
'a measurement. ^ , 

(2) Log this point and test equipment you would use on. the Tedt 
Performance She*et . ^1 , 

(3) Examine Equipment Test Point" arid Tube Fin D^ta Sheet for point 

** ' * • .-■....-■.-t^ ' 

you have selected ai\d observe data given.'' Determine whether 

t . - 

' the reading given is, normal or abnormal dnd log on T^eet 

' ' t» . . ' ^ ^ • 

Performance Shefet. 

* V . . ' : . 

(4) With this new information repeat the process- until you feel 

' ' " :r^' • ; <: , ^l ^ . ^ : 

you cam ^isolate no'furthe^r vith the information given.* 

' * . » - - . 4 . • ' . 

(5) Sow Ipgthe suspected stage or stages and the tubes you wish. 

. * /to chedk next. . - / , 

(6) Turn in ^Twt Perfonnaince Sheet to Test Administrator. 



^4 







&PART II 

> 








TfesT Performance sheet 




Problem No. 5 
Point Measw 






Tprhnir.lan ID No.: 


I 

ed 


/ Test Equipment. 
You Would Use 


Is Reaaing .Normal 
or Abnormal 








• 






^ *■ 


^ ♦ 


\- . ■ , - _ 


-i ■ - 


, 


> 






/ 


• 

ft 


• ' . ■ - " . ■ m 
. - r. ■ ' . ' ■ ■ ■ 




I J ' ' 


' — >^ — ^ — 


: — 









Suspect Stjfiigclor Stiages^ 



yubes you wish to 6heck^ 



0 • 
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PART II 
Problem No. 

VISUAL INDICATIONS 



DOPPLER TEST HARNESS METER SWITCH READINGS 
Position 
115,400 CPS 
+28V 
+500V 
+130V 
-425V. 



Reading* 
Normal 
formal 
Normal 
Normal 
Normal 



FREQUE^qCY TRACKER TEST PANEL METER 'SWITCH READINGS 



Position 
XTAL A 
XTAL B 
IF A 
IF B 
COMP 



Reading. 

Normal 

Normal 

Normal ' 

Normal 

Abnormal 



. DRIFT ANGLE/GROUNDSBEED INDICATOR 
Drift Angle/ Indicator "Groundspeed Counter. 

Normal, b*ut may be slewed Abnormal, but may be slewed 
Memory Light 

Remains on • 



• I 



FREQUENCY TRACKER TUBE PIN READINGS ; 



•njBE 


1 


Z 




A 
*♦ 


TU 


BE PIN 
5 


7 ' 


8 


9 


10 


irb.oi(v) 


3.I5VAC 


I.AVAC 


3aVAC 


7 • 2VAC 




120VDC 


2,65VAC 


4,3VDC 


3a5VAC 




(R) 


150 


13 


A70K 


^A7K 


0 


15 


' lOOK ^ 


1,5K 


150 




V6702(V) 


I02VDC 
2.3VAC 


3.0VAC 


2iaVDC 
2 • 6VAC 


3a5VAC 


3.15VAC 


75VDC 

X « O V A\/ 


OaBVAC 


laVDC 


3.15VAC 






4.7K 


, A75K 


5.2K 


150 


150 


lOK 


43QK 


220 


150 




V6703(V) 


I02VDC 
2,2VAC 


3.0VAC 


2I,IVPC 
2 . 6VAC 


3,I5VAC 


3.15VAC 


T-SVDC 

' 1 7UAP 


0a6VAC 


1,1VDC 


aasvAc I — 




4.7K 


A75K 


• 5.2K 


150 


150 


lOK 


430K 


220 


• 15Q 1 — 


y6704(V) 


I20VDC 
2,5VAC 


2. A VAC 


^5.8VDC 


3.I5VAC 


3.15VAC 


120VDC 
Z « DVAV' 


2,35VAC 


5-8VDC 


3a5VAC J — 


\»*/ 




lOOK 


I.5K 


150 . 


150 


6^ 


IpOK- 


;i-5K 


150 1 


y6705(V). 


I20VDC 
2\7VAC 


2.2VAC 


5,8VDC 


3a5VAC 


3.15VAC 


120VDC 

Z « / VAO ^ 

6 \ " 


2,35VAC 
lOOK . 


5-8VDC 


3a5VAC ; 


W 


6 


lOOK 


I.5K 


150 


150 


. 1*5K 


150 1 


V6706(V) 




w^A3VDC 


-IA7VDC 






"~ 7--" - 


0,95VAC 


-1A3VDC 


1 


(R) 




2•2K^ 


0-lOK 


0 ^/ 




330K 


, IMEG* 


2.2K- 




V6707(V) 




-U3VDC 


-U7VDC 








• 0,95V^G 


-1A3VDC ; 




. (R) 




2,2K 








330KV ^ 




2,2K 




V ^ .8(V) 


9AVDC 
2. A VAC 


I.6VAC 


•.^.4VDC^ 


3-15^'AiS: 




9AVDQ 
2,AVAC ► 


1-^VAC; 


A.AVDC 


3a5VAC ^ 


(R) 


lOK 


2A 


I.'5K 


150- \. 




lOK j 




: 1-5K 


150 


V6709 (V) 


I20VDC 
0.8AVAC 


a.6VAC 


5,8VDC 


3a5VAC 




120VDC 
0* 84 VAC 


; 1.65VAQ 


6.1VDC 


.3a3VAC 


1 

1 


tR) 


15 


24 


3,3-A,.3K 


: 150 


150 ' 






A,3-3.3K 


150 




V6710(V) 


I20VDC . 


,I.6^VA( 


: 50VDC 
1,5VAC 


3-15VAC 


|3,15VAC 


il20VDC 


1-65VAC 


. 50V|^. 
I.5VAC 


1 3. 15 VAC 




^ (R) 


0 


57K 


10. 5K 


150 ;^ 


1 


0 


57K c 


10.5)f 


j 150 





NOTES: 



a\d a 3^.0 kc, 0.16 volts mc signal to channel B (pin 7-V6703) 

2. ^ All voltages measured with respect to gtpund 1 

3. All voltages measured" with tubes in place and POWER ON. | 

4. Air resistance measured in ohms vlt.h reflect to ground and tubes removed 
5-. Resistan^efieisurements are made with pins \ and 6. of P6701 cWtPd to 2 



shorted, to ground. 

It may be necessary to reverse the ohmeter leads to obtain these readings • 
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FREQUENCY TRACKER T^T POINT READTKV.S 
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APPENDIX C • 
PART III. TEST MATERIALS 
Isolation to Piece/Part *tevel 
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• . *. » PART III 

^ TEST /S^J^Cy- TEST INSTRUCTIONS 

Part III: I-solation to the -Piece/Part . . . 

INSTRUCTIONS: . ' 

a. Ensure 'that you have a TM and the following information sheets. 

(1) Equipment Visual Indications - 

-« - 

(2) Equipment Test Point Data Sbee'ts 

(3) Equipment Tube Pin Data 

(4) Part III Test Performance Sheet 

(5) Piece/Part Data 

b. Enter your assigned identification and problem number on. the Test 
Performance Sheet. ./i 

c. ^^^roblem has been isolated to a faulty module. Your job^ is to 

Isolate this problem to a faulty part. Piece /part data Is' provided 
on the faulty module. ^ ^ 

d. You will be simulating roeasurefit^ts on the equipments as follows: 

(1) Use visual indications and TO to decide where or what part '.you 
wish to measure. Assujne all vacuum tubes are good. 

(2) Log this point or part ^nd the test equipment you would use on 
the Test Performance Sheet. 

(3) Examine correspon'ding Data Sheets and observe data given 
Determine? whether the reading glyen is normal or abnormal and.* 
log on Test Performance Sh^et. 

(4) With this new information repeat the process until you have 



ERIC 



isolated to the faulty part. - 1 

(5) /Now log the suspected* part on the Test- Performapce' Sheet^.i, 

C6) Turn In. Test Performance" Sheet to Test Administrator. \ r 

o '^^^ ■ 74 ^ ' 



PART III 
TEST PERFORMANCE SHEET • 



0 



Problem No. : 



Technician ID No.: 



Point Measured 
or Part Checked 



Test Equipment 
You Would Use^ 



Is Reading Normal 
or Abnormal 



Vj^ Suspect Part 



73 
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PART 111 



PART 



C7901 



C7902 



C7903 



C7905- 



V 



C7908 



C7913 



C7914 



C7917 



C7919 



-.C7920 



CR790r 



CR,7902 



KKADTVG 




K7901-13/12 
K7901-13/14 



K7901-1/2 



H7901 



R7902 



R7903 



R7904 



FREQUENCY TRAQKER 

Piece/Part Data, 
Problem No,. 2. 




R79C5 



R7906 



-R7907 




PART 



R7908 



R7909 



R7910 



R7911 



R7912 



R7913 



R7925 



R7926 



R7939 



R79AO 



«il';''#>,-\' ,v « ? 







T7901-I/2 
T79ni-3//. 




Sheet i 
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